We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks.

The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery. The position is full-time and onsite in Toronto, ON

.

Key Requirements

- 5+ years of experience in Quality Engineering, QA, SDET, or Test Automation .
- 5+ years of programming experience with Java, Python, Groovy, or similar languages .
- Strong experience building and maintaining test automation frameworks .
- Hands-on experience with Selenium, Playwright, Cucumber/Gherkin, or RestAssured .
- Experience with API, database, integration, regression, and microservices testing .
- Experience with JUnit, TestNG, ReadyAPI, qTest, and JMeter .
- Hands-on experience testing AI/ML, Generative AI, conversational AI, or LLM-based applications .
- Knowledge of AI quality concepts such as prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias,



and response consistency .
- Experience with CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing .
- Robust analytical, problem-solving, communication, and documentation skills.

Key Responsibilities

- Develop and execute comprehensive quality strategies, test plans, test cases, and test data .
- Design, develop, and maintain scalable test automation frameworks .
- Create automated tests for AI/ML and Generative AI features .
- Evaluate AI responses for accuracy, relevance, safety, hallucination, reliability, and latency .
- Test APIs, microservices, databases, integrations, data pipelines, and third-party services .
- Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage.
- Integrate automated testing and AI quality checks into CI/CD pipelines .
- Identify, document, prioritize, and track defects through resolution.
- Analyze test coverage, quality metrics, AI evaluation results, and production feedback.
- Collaborate with Development, DevOps, Product, and QE teams to improve product quality and release readiness.

📌 Senior Software Quality Engineer – AI (Toronto)
🏢 PeoplePilot
📍 Toronto