Join Dialpad as an AI Evaluation Engineer, directly impacting AI-driven customer experiences in Vancouver. Your role will focus on developing LLM-judge metrics and evaluation strategies for agentic systems.

This position requires a driven individual with a strong background in QA, test engineering, and applied ML quality assurance. You will co-own various aspects of AI evaluation, including validation strategies and error analysis to ensure product readiness. Collaborate across teams to refine our AI systems and support their deployment in a competitive market.

Key Responsibilities:



• Design and implement validation strategies for AI workflows • Build and refine regression evaluations and red teaming analyses • Collaborate on LLM-judge metric calibration and prompt strategies • Manage data annotation jobs for timely evaluation datasets • Investigate bugs and recommend escalation or further analysis

Requirements: • Bachelor’s or Master’s degree in Computer Science or similar • Minimum of 3 years of experience with AI product evaluations • Proficient with NLP, speech, and AI evaluation datasets • Solid analytical skills for identifying quality issues • Effective collaboration with technical teams and stakeholders

Drive AI innovation at Dialpad by applying your skills to evaluate and enhance next-generation customer solutions. #J-18808-Ljbffr

📌 AI Evaluation Engineer Role at Dialpad (Winnipeg)
🏢 Doist
📍 Winnipeg