Lead AI evaluation strategy from Canada, applying automated testing and model governance to improve agent reliability across enterprise automations.
As Lead Product Manager, Agent Runtime and Evals, you will shape evaluation infrastructure for an AI platform. Your scope includes golden datasets, data labeling, automated judges, regression testing, quality gates, and human-in-the-loop evaluation. You will also guide runtime capabilities that help non-technical users test, publish, monitor, and improve agents.
Key Responsibilities: Define evaluation and model governance frameworks for production readiness
Establish mechanisms for agents assessing other agents
Lead testing, quality measurement, and continuous improvement capabilities
Shape agent memory, learning, and self-improvement features




Partner with engineering and design on intuitive configuration tools
Requirements: Seven-plus years managing products in B2B or enterprise SaaS
Three or more years working in AI-focused product roles
Robust understanding of golden datasets and data labeling
Experience with automated judges, regression testing, and human evaluation
Ability to explain complex decisions to technical and executive audiences
Bring product leadership to AI evaluation, model governance, agent testing, and measurable quality across large-scale automation environments.

#J-18808-Ljbffr

📌 Lead Product Manager AI Evaluations (Manitoba)
🏢 Jobgether
📍 Manitoba