Lead AI evaluation strategy from Canada, applying automated testing and model governance to improve agent reliability across enterprise automations.As Lead Product Manager, Agent Runtime and Evals, you will shape evaluation infrastructure for an AI platform. Your scope includes golden datasets, data labeling, automated judges, regression testing, quality gates, and human-in-the-loop evaluation. You will also guide runtime capabilities that help non-technical users test, publish, monitor, and improve agents.Key Responsibilities:

Define evaluation and model governance frameworks for production readinessEstablish mechanisms for agents assessing other agentsLead testing, quality measurement, and continuous improvement capabilitiesShape agent memory, learning,



and self-improvement featuresPartner with engineering and design on intuitive configuration toolsRequirements:

Seven-plus years managing products in B2B or enterprise SaaSThree or more years working in AI-focused product rolesStrong understanding of golden datasets and data labelingExperience with automated judges, regression testing, and human evaluationAbility to explain complex decisions to technical and executive audiencesBring product leadership to AI evaluation, model governance, agent testing, and measurable quality across large-scale automation environments.

📌 Lead Product Manager Ai Evaluations (Winnipeg)
🏢 Jobgether
📍 Winnipeg