Elevate customer experience at Dialpad as an AI Evaluation Engineer, focusing on agentic AI systems in Vancouver. Ensure quality and reliability across NLP and speech workflows while collaborating with cross-functional teams. As a key player in our AI Evaluation team, you will co-own metric development and calibration, curate evaluation datasets, and lead structured error analysis.

Your contributions will directly influence readiness decisions for our advanced AI voice and chat solutions. Engage in designing validation strategies and executing A/B comparisons to ensure optimal product performance. Key Responsibilities:
Design validation strategies for NLP and speech workflows
Build and enhance regression evaluations and A/B tests




Co-own LLM-judge metric development and calibration
Create and monitor data annotation jobs for evaluations
Investigate bugs and triage issues effectively Requirements:
Bachelor’s or Master’s in Computer Science or related field
3+ years in QA, test engineering, or model evaluation
Experience with AI system test strategies
Solid analytical skills for quality pattern identification
Team-oriented skills in technical documentation Harness your skills in QA and AI validation to drive Dialpad's innovation forward.

📌 Ai Evaluation Engineer At Dialpad Kitchener
🏢 Dialpad
📍 Kitchener