Join Dialpad as an AI Evaluation Engineer, directly impacting AI-driven customer experiences in Vancouver.

Your role will focus on developing LLM-judge metrics and evaluation strategies for agentic systems. This position requires a driven individual with a solid background in QA, test engineering, and applied ML quality assurance. You will co-own various aspects of AI evaluation, including validation strategies and error analysis to ensure product readiness. Collaborate across teams to refine our AI systems and support their deployment in a competitive market. Key Responsibilities:

- Design and implement validation strategies for AI workflows
- Build and refine regression evaluations and red teaming analyses
- Collaborate on LLM-judge metric calibration and prompt strategies




- Manage data annotation jobs for timely evaluation datasets
- Investigate bugs and recommend escalation or further analysis Requirements:
- Bachelor’s or Master’s degree in Computer Science or similar
- Minimum of 3 years of experience with AI product evaluations
- Proficient with NLP, speech, and AI evaluation datasets
- Strong analytical skills for identifying quality issues
- Effective collaboration with technical teams and stakeholders Drive AI innovation at Dialpad by applying your skills to evaluate and enhance next-generation customer solutions.

📌 AI Evaluation Engineer Role at Dialpad (Kitchener)
🏢 Doist
📍 Kitchener