Join Dialpad as an AI Evaluation Engineer, directly impacting AI-driven customer experiences in Vancouver. Your role will focus on developing LLM-judge metrics and evaluation strategies for agentic systems.

This position requires a driven individual with a robust background in QA, test engineering, and applied ML quality assurance. You will co-own various aspects of AI evaluation, including validation strategies and error analysis to ensure product readiness.

Collaborate across teams to refine our AI systems and support their deployment in a competitive market.

Key Responsibilities:
Design and implement validation strategies for AI workflows
Build and refine regression evaluations and red teaming analyses
Collaborate on LLM-judge metric calibration and prompt strategies




Manage data annotation jobs for timely evaluation datasets
Investigate bugs and recommend escalation or further analysis

Requirements:
Bachelor’s or Master’s degree in Computer Science or similar
Minimum of 3 years of experience with AI product evaluations
Proficient with NLP, speech, and AI evaluation datasets
Solid analytical skills for identifying quality issues
Effective collaboration with technical teams and stakeholders

Drive AI innovation at Dialpad by applying your skills to evaluate and enhance next-generation customer solutions.
J-18808-Ljbffr

📌 Ai Evaluation Engineer Role At Dialpad Kitchener
🏢 Doist
📍 Kitchener