We are seeking a

Senior Software Quality Engineer – AI Focused

to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks. The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery. The position is

full-time and onsite in Toronto, ON .

Key Requirements

5+ years of experience in

Quality Engineering, QA, SDET, or Test Automation .

5+ years of programming experience with

Java, Python, Groovy, or similar languages .

Robust experience building and maintaining

test automation frameworks .

Hands-on experience with

Selenium, Playwright, Cucumber/Gherkin, or RestAssured .

Experience with

API, database, integration, regression, and microservices testing .

Experience with

JUnit, TestNG, ReadyAPI, qTest, and JMeter .

Hands-on experience testing

AI/ML, Generative AI, conversational AI, or LLM-based applications .

Knowledge of AI quality concepts such as

prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias,



and response consistency .

Experience with

CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing .

Strong analytical, problem-solving, communication, and documentation skills.

Key Responsibilities

Develop and execute comprehensive

quality strategies, test plans, test cases, and test data .

Design, develop, and maintain scalable

test automation frameworks .

Create automated tests for

AI/ML and Generative AI features .

Evaluate AI responses for

accuracy, relevance, safety, hallucination, reliability, and latency .

Test

APIs, microservices, databases, integrations, data pipelines, and third-party services .

Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage.

Integrate automated testing and

AI quality checks into CI/CD pipelines .

Identify, document, prioritize, and track defects through resolution.

Analyze test coverage, quality metrics, AI evaluation results, and production feedback.

Collaborate with

Development, DevOps, Product, and QE teams

to improve product quality and release readiness.

#J-18808-Ljbffr

📌 Senior Software Quality Engineer – AI (Winnipeg)
🏢 PeoplePilot
📍 Winnipeg