Advance Dialpad's AI Evaluation team as an AI Evaluation Engineer, contributing to cutting-edge solutions in Vancouver. Focus on LLM-judge metrics, validation strategies, and cross-functional collaboration for impactful AI assessments. In this role, you will co-own the evaluation coverage for our Agentic AI systems while enhancing NLP, speech workflows, and structured error analysis.

This position requires a detail-oriented qualified with experience in QA, test engineering, and model evaluation. You will drive release-readiness decisions by developing regression evaluations and collaborating with various teams. Key Responsibilities:
Design validation strategies for NLP and speech workflows
Build and improve regression evaluations and A/B comparisons




Co-own LLM-judge metric development and prompt refinement
Monitor data annotation jobs for evaluation datasets
Investigate bugs and decide on escalation or follow-up analysis Requirements:
Bachelor’s or Master’s degree in Computer Science or related field
3+ years in QA or model evaluation for AI products
Experience with NLP or advanced AI systems
Solid analytical skills for investigating quality issues
Cooperative skills for cross-functional communication Join Dialpad in enhancing AI-driven customer experiences through your skills in evaluation and collaboration.

📌 Ai Evaluation Engineer At Dialpad Kitchener
🏢 Doist
📍 Kitchener