Contract Duration: May 4, 2026 to November 4, 2026

Overview

We are seeking an experienced AI / ML QA Engineer to support testing and validation of advanced machine learning and generative AI systems. This role focuses on ensuring model quality, reliability, and responsible AI practices across a range of applications including NLP, classification, recommendation systems, and large language models.

Key Responsibilities

Functional and Model Testing

- Design and execute test cases for AI and ML models including NLP, classification, recommendation, and generative AI systems
- Validate outputs for accuracy, consistency, and expected behavior across diverse datasets and inputs
- Perform regression testing following model updates, retraining cycles, or data changes

Prompt and LLM Testing

- Develop and maintain prompt test libraries for LLM based features
- Evaluate responses for relevance, tone, factual accuracy, and alignment with business rules
- Identify hallucinations, response drift, and edge case failures

AI Safety and Red Teaming

- Conduct adversarial testing to uncover vulnerabilities such as prompt injection and jailbreak scenarios
- Assess outputs for bias, fairness, and compliance with responsible AI standards
- Document risks and collaborate with model teams on remediation strategies

Test Automation and Tooling

- Build and maintain automated test suites using Python based frameworks




- Track and report quality metrics, model performance benchmarks, and defect trends

Collaboration and Documentation

- Work closely with data scientists, ML engineers, and product teams
- Develop explicit test plans, defect reports, and evaluation summaries
- Contribute to QA standards and best practices for AI systems

Required Skills

- Hands on experience testing AI and ML models including NLP, classification, and generative AI
- Strong experience with LLM testing including prompt engineering and evaluation of hallucinations, bias, and response quality
- Proficiency in Python and test automation frameworks
- Experience working with CI CD pipelines
- Solid understanding of regression testing

Nice to Have

- Experience in banking or financial services
- Exposure to AI safety and red teaming techniques such as prompt injection and jailbreak testing
- Knowledge of responsible AI and model governance principles
- Familiarity with LLM evaluation tools and frameworks
- Experience testing recommendation or ranking systems
- Exposure to ML pipelines and MLOps workflows
- Understanding of bias and fairness metrics

Note: We use AI tools to: obtain basic information, detect plagiarism, false employment history or references, categorize your skills, and do an initial match with job posting.

#J-18808-Ljbffr

📌 AI / ML QA Engineer (Toronto)
🏢 Infotek Consulting
📍 Toronto