We are seeking a Senior Software Quality Engineer – AI Focused to drive quality, reliability, and automation for AI-enabled software products. The role will collaborate with Development, DevOps, Product, and QE teams to design robust testing strategies and scalable automation frameworks. The engineer will test AI/ML and Generative AI features, APIs, microservices, integrations, and data pipelines while ensuring reliable and secure product delivery.

The position is full time and onsite in Toronto, ON.

Key Requirements

- 5+ years of experience in Quality Engineering, QA, SDET, or Test Automation.
- 5+ years of programming experience with Java, Python, Groovy, or similar languages.
- Strong experience building and maintaining test automation frameworks.
- Hands-on experience with Selenium, Playwright, Cucumber/Gherkin, or RestAssured.
- Experience with API, database, integration, regression, and microservices testing.
- Experience with JUnit, TestNG, ReadyAPI, qTest, and JMeter.
- Hands-on experience testing AI/ML, Generative AI, conversational AI, or LLM-based applications.
- Knowledge of AI quality concepts such as prompt evaluation, accuracy, relevance, grounding, hallucination detection, toxicity, bias,



and response consistency.
- Experience with CI/CD, Agile, BDD, TDD, ATDD, and risk-based testing.
- Strong analytical, problem-solving, communication, and documentation skills.

Key Responsibilities

- Develop and execute comprehensive quality strategies, test plans, test cases, and test data.
- Design, develop, and maintain scalable test automation frameworks.
- Create automated tests for AI/ML and Generative AI features.
- Evaluate AI responses for accuracy, relevance, safety, hallucination, reliability, and latency.
- Test APIs, microservices, databases, integrations, data pipelines, and third-party services.
- Create AI evaluation datasets, expected behaviors, quality thresholds, and regression coverage.
- Integrate automated testing and AI quality checks into CI/CD pipelines.
- Identify, document, prioritize, and track defects through resolution.
- Analyze test coverage, quality metrics, AI evaluation results, and production feedback.
- Collaborate with Development, DevOps, Product, and QE teams to improve product quality and release readiness.

📌 Senior Software Quality Engineer – AI (Toronto)
🏢 PeoplePilot
📍 Toronto