- The AI Evaluation Engineer ensures AI agent solutions are accurate, reliable, protected, and production-ready within regulated banking environments. The role provides objective, evidence-based evaluation of AI agent behaviour against defined business, quality, risk, performance, and regulatory criteria, helping ensure AI solutions deliver consistent, trusted outcomes in production

- Working within the Reliability Testing phase of the AIOS (Zafin’s AI Operating System) delivery lifecycle, the role designs, executes, and leads evaluation activities that validate AI agent behaviour across the full development lifecycle. This includes developing meaningful evaluation scenarios, identifying defects and failure modes, monitoring quality across releases, and providing actionable feedback that continuously improves AI agent reliability. The role helps operationalize AIOS’s principle of reliability first, velocity second through disciplined evaluation and objective production-readiness decisions
- The AI Evaluation Engineer works closely with Agent Engineers, Industry Consultants, Agent Architects, AI Knowledge & Governance, Product, and Delivery teams to ensure evaluation reflects intended business logic, regulatory requirements, technical standards, and real-world operating conditions.

Depending on experience and level, the role may also lead evaluation activities, coach other evaluation engineers, improve evaluation practices, and drive quality improvements across multiple AI agent initiatives

- Ultimately, the role helps ensure AI agent capabilities earn and maintain customer trust by delivering consistent, reliable, and explainable outcomes in production





- Design and execute structured evaluation scenarios that validate AI agent accuracy, reliability, safety, compliance, and business outcomes

- Validate AI agent behaviour against approved business rules, policies, technical requirements, source knowledge, and expected outcomes

- Conduct regression evaluation across releases and monitor behavioural drift, performance degradation, and newly introduced failure modes

- Identify, document, prioritize, and track quality issues, defects, and production risks

- Support production-readiness decisions through objective evaluation evidence and recommendations

- Analyze evaluation results to identify root causes, recurring quality trends, and opportunities to improve prompts, workflows, knowledge, integrations, and engineering practices

- Maintain reusable evaluation scenarios, benchmark datasets, expected outcomes, regression suites, and supporting evidence

- Contribute to continuous improvement of evaluation methodologies, automation, tooling, and engineering feedback loops

- Support investigation of production issues and validate corrective actions

- Partner with Agent Engineering, Industry Consultants, Agent Architects, AI Knowledge & Governance, Product, and Delivery teams to ensure evaluation reflects business requirements and production expectations





- Participate in production-readiness reviews, release planning, and AI agent optimization activities

- Communicate evaluation findings, quality risks, and recommendations clearly to technical and business stakeholders

- Depending on experience and level, you may also:

- Lead evaluation activities across one or more AI agent initiatives or squads

- Review evaluation approaches, production-readiness recommendations, and quality evidence produced by other Evaluation Engineers

- Coach and mentor less experienced Evaluation Engineers, supporting technical growth and consistent evaluation practices

- Coordinate evaluation priorities across multiple concurrent initiatives and support delivery planning

- Drive improvements in evaluation tooling, automation, benchmark management, CI/CD quality integration, and operational effectiveness

- Analyze systemic quality trends and lead continuous improvement initiatives across multiple AI agent capabilities

- Partner with Delivery and Engineering leadership to improve quality outcomes, operational consistency, and AI agent reliability

Benefits

- Flexible work arrangements - includes opportunities for hybrid work, flexible hours, flexible locations, and various types of paid leave

- Meaningful recognition - we truly value our team members’ contributions and ensure that their achievements are properly recognized and rewarded through areas such as incentive opportunities

- Ownership - we offer a stock option plan that is exclusively for employees who share our sense of purpose and create value that keeps us on our A-game

📌 AI Evaluation Engineer (Toronto)
🏢 Zafin
📍 Toronto