About the Role & TeamIDC is building the next generation of AI-powered intelligence platforms that transform how technology decisions get made. Our platform re-imagines the way decision-makers discover and interact with trusted research and data, and is foundational to IDC's future.What You’ll DoDesign and build the evaluation infrastructure that ensures the platform's AI systems produce accurate, well-sourced, high-quality responsesBuild automated test suites that validate answer quality across agent pipeline changesDevelop regression detection systems that catch quality degradation before it reaches usersCreate evaluation frameworks that measure response accuracy, citation correctness, and source qualityWork closely with the product team to translate quality criteria into measurable, automatable test scenariosBuild cost and latency monitoring that tracks the operational efficiency of AI pipeline executionDefine evaluation standards and practices that scale as the platform and team growWhat You Bring6+ years of software engineering experience, with significant work in testing infrastructure, ML evaluation, or quality systemsExperience building evaluation or testing frameworks for LLM-based or ML-based systemsUnderstanding of how to measure response quality for generative AI: accuracy, groundedness, citation correctness, relevanceProficiency in PythonAbility to operate independently and define your own roadmap. You will be the first hire in this functionExperience working at the intersection of engineering and product,



translating qualitative quality criteria into quantitative measurementsExperience with LLM evaluation frameworks (e.G., RAGAS, DeepEval, or custom)Familiarity with LLM observability tools (e.G., Langfuse, LangSmith, Weights & Biases)Background in statistical methods for quality measurement (significance testing, distribution analysis)Experience building A/B testing or experimentation infrastructureBackground in search relevance evaluation or information retrieval metricsWhy This Role Stands OutAt IDC, your work helps shape how the world understands technology and where it goes next. You collaborate with curious, high-caliber colleagues who value rigor, integrity, and shared success. As the premier global provider of trusted technology intelligence, IDC equips business and technology leaders with the evidence they need to make confident decisions. Our insights inform strategy, investment, and innovation across industries and regions. Recognized by IIAR as Analyst Firm of the Year for five consecutive years, IDC sets the standard for credibility and impact. With more than 1,000 analysts worldwide and a truly global perspective, we combine deep expertise with practical relevance. Here, your ideas matter, your voice is heard, and your contributions provide the insights leaders rely on every day. It is meaningful work,



backed by a culture that supports growth, collaboration, and long‑term career development with a globally respected brand.What We Offer15 vacation days per year (increases with tenure; carryover allowed)10 paid sick days per year1 week paid new parenting leaveFlexible work options (remote, part‑time, flexible schedule)Health, dental, vision, and paramedical coverage for you and your family$1,600 annual healthcare spending accountEmployee Assistance Program for counseling and supportBest Doctors medical second opinionsLife, AD&D, and long‑term disability insuranceRetirement savings plan with company match (up to 4% of salary)$75/month technology allowance for home office or phone expensesCompany‑paid cell phone planCompensation TransparencyAt IDC, we are committed to fair and equitable pay practices. Employees are compensated equitably for their work, aligned with their skills and experience. Salary and incentive structures are determined through a rigorous process that considers experience, education, certifications, role‑specific requirements, internal equity, and verified Canada market data from an independent third‑party partner. The expected total annual compensation, depending on location and experience, is between 100,000 CAD – 143,000 CAD, inclusive of base salary and variable compensation.Equal Opportunity EmployerIDC is committed to providing equal employment opportunities for all qualified persons. Employment eligibility verification required. We participate in E‑Verify.#J-18808-Ljbffr

📌 Ai Engineer – Ai Observability And Quality - C$100,000 - C$143,000 A Year (Toronto)
🏢 IDC
📍 Toronto