Lead AI evaluation strategy from Canada, applying automated testing and model governance to improve agent reliability across enterprise automations.

As Lead Product Manager, Agent Runtime and Evals, you will shape evaluation infrastructure for an AI platform. Your scope includes golden datasets, data labeling, automated judges, regression testing, quality gates, and human-in-the-loop evaluation. You will also guide runtime capabilities that help non-technical users test, publish, monitor, and improve agents.

Key Responsibilities

Define evaluation and model governance frameworks for production readiness

Establish mechanisms for agents assessing other agents

Lead testing, quality measurement, and continuous improvement capabilities

Shape agent memory, learning, and self-improvement features





Partner with engineering and design on intuitive configuration tools

Requirements

Seven-plus years managing products in B2B or enterprise SaaS

Three or more years working in AI-focused product roles

Solid understanding of golden datasets and data labeling

Experience with automated judges, regression testing, and human evaluation

Ability to explain complex decisions to technical and executive audiences

Bring product leadership to AI evaluation, model governance, agent testing, and measurable quality across large-scale automation environments.

📌 Lead Product Manager AI Evaluations (Winnipeg)
🏢 Jobgether
📍 Winnipeg