Lead AI evaluation strategy from Canada, applying automated testing and model governance to improve agent reliability across enterprise automations.

As Lead Product Manager, Agent Runtime and Evals, you will shape evaluation infrastructure for an AI platform. Your scope includes golden datasets, data labeling, automated judges, regression testing, quality gates, and human-in-the-loop evaluation. You will also guide runtime capabilities that help non-technical users test, publish, monitor, and improve agents.

Key Responsibilities:

- Define evaluation and model governance frameworks for production readiness
- Establish mechanisms for agents assessing other agents
- Lead testing, quality measurement, and continuous improvement capabilities
- Shape agent memory, learning, and self-improvement features




- Partner with engineering and design on intuitive configuration tools

Requirements:

- Seven-plus years managing products in B2B or enterprise SaaS
- Three or more years working in AI-focused product roles
- Solid understanding of golden datasets and data labeling
- Experience with automated judges, regression testing, and human evaluation
- Ability to explain complex decisions to technical and executive audiences

Bring product leadership to AI evaluation, model governance, agent testing, and measurable quality across large-scale automation environments.

📌 Lead Product Manager AI Evaluations (Canada)
🏢 Jobgether
📍 Canada