We are seeking a Senior Manager to join the Model Risk Management (MRM) function and provide independent validation, challenge, and governance oversight of Machine Learning (ML), Artificial Intelligence (AI), and Generative AI (GenAI) models used across the organization.

The successful candidate will evaluate traditional predictive models, Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) applications, and advanced analytics solutions to ensure they are conceptually sound, fit for purpose, compliant with regulatory expectations, and appropriately governed throughout their lifecycle.

This role requires a robust combination of quantitative modelling expertise, model validation experience, AI governance knowledge, and familiarity with emerging risks associated with Generative AI.

Key ResponsibilitiesAI & ML Model Validation

- Perform independent validation of AI, ML, and statistical models in accordance with Model Risk Management frameworks.
- Assess conceptual soundness, methodology, assumptions, data quality, performance, stability, and model limitations.
- Review model development documentation, implementation processes, monitoring controls, and governance artifacts.
- Develop and evaluate challenger models where appropriate.
- Prepare validation reports, findings, and recommendations for model owners, risk committees, and senior stakeholders.

Generative AI & LLM Validation

- Independently review and challenge Generative AI solutions utilizing LLMs, RAG architectures, vector databases, embeddings, and semantic search capabilities.
- Assess:
- Prompt design and controls
- Retrieval effectiveness
- Hallucination risk
- Groundedness and faithfulness
- Explainability and transparency
- Bias and fairness risks
- Output consistency and robustness
- Evaluate retrieval and embedding performance using quantitative and qualitative techniques, including recall, precision, relevance,



and human-in-the-loop assessments.
- Conduct benchmarking against alternative models and industry standards.

AI Risk Assessment & Governance

- Identify and assess emerging AI risks including hallucinations, model drift, prompt injection, data leakage, bias, and inappropriate model behavior.
- Evaluate compliance with internal Model Risk Management policies and external regulatory expectations.
- Support Responsible AI initiatives by assessing fairness, explainability, accountability, transparency, and governance controls.
- Contribute to the development of AI validation standards, testing methodologies, and governance frameworks.

AI Testing & Monitoring

- Execute scenario testing, adversarial testing, sensitivity analysis, and outcome analysis for AI and GenAI applications.
- Review production monitoring frameworks covering:
- Model performance
- Drift detection
- Retrieval quality
- Hallucination rates
- User feedback and operational metrics
- Assess monitoring methodologies and thresholds used to identify model degradation and retraining requirements.

Required Experience

- 7+ years of experience in Model Risk Management, Model Validation, Data Science, Quantitative Analytics, or Machine Learning.
- Experience validating predictive models within banking, asset management, insurance, or other regulated industries.
- Experience reviewing and assessing Machine Learning models including:
- Logistic Regression
- Random Forest
- Gradient Boosting / XGBoost
- Ensemble Models
- Hands-on experience evaluating Generative AI and LLM-based applications.
- Strong understanding of Model Risk Management principles and regulatory expectations.

Pay: $85.00 per hour

Ability to commute/relocate:

- Toronto, ON M5C 1H6: reliably commute or plan to relocate before starting work (required)

Experience:

- AI: 2 years (required)
- Governance and Risk : 3 years (required)

Work Location: Hybrid remote in Toronto, ON M5C 1H6

📌 AI Validation and Testing Consultant (Canada)
🏢 Allevar
📍 Canada