Toronto, Ontario M5V 3L9 Posted October 9th, 2026

Job Type: Full Time

Job Category: IT

Quality Engineer / SDET – AI Chatbot Validation

Primary Skills: AI/LLM Chatbot Testing, Test Automation (Python/Selenium/Playwright), API Testing & Quality Assurance

Location: Toronto, ON

Work Model: Onsite

Job Type: Full Time

We are seeking an experienced Quality Engineer (QE) / Software Development Engineer in Test (SDET) to validate AI-powered chatbots and conversational applications. The role focuses on ensuring chatbot responses are accurate, relevant, complete, consistent, and aligned with expected behavior.

The ideal candidate will have strong QA and test automation experience, familiarity with AI/LLM-based applications, and the ability to identify hallucinations, incorrect responses, formatting issues, and knowledge gaps. The candidate will collaborate with developers, AI engineers, and content teams to improve chatbot quality and reliability throughout the development lifecycle.

Key Responsibilities

- Validate chatbot responses for accuracy, relevance, completeness, consistency, and contextual correctness.
- Design, develop, and execute test scenarios, test cases, and regression test suites for AI-powered chatbots.
- Identify and document hallucinations, incorrect answers, formatting defects, and knowledge gaps.
- Perform chatbot conversation analysis and recommend improvements to response quality and user experience.
- Develop and maintain automated test scripts and frameworks for chatbot validation where feasible.
- Conduct API testing and validate integrations with supporting services and knowledge sources.
- Collaborate with developers, AI engineers, and content teams to investigate and resolve defects.
- Track, document,



and report defects, test results, quality metrics, and testing progress.
- Support User Acceptance Testing (UAT), release validation, and production quality monitoring.
- Apply prompt engineering concepts to design effective test scenarios and evaluate chatbot behavior.
- Participate in Agile ceremonies and contribute to continuous quality improvement.

Required Skills & Experience

- 5 years of experience in Quality Assurance, SDET, software testing, or test automation.
- Experience testing AI, LLM, Generative AI, chatbots, virtual assistants, or conversational platforms.
- Understanding of prompt engineering concepts and AI response evaluation.
- Experience with API testing and test automation frameworks.
- Hands-on experience with Python, Selenium, Playwright, PyTest, or similar testing tools.
- Experience with Jira, Confluence, GitHub, and Agile delivery methodologies.
- Solid analytical, troubleshooting, and problem-solving skills.
- Ability to identify response inconsistencies, incorrect outputs, and potential AI hallucinations.
- Excellent communication, defect reporting, and test documentation skills.

Preferred Skills

- Experience developing automated validation frameworks for AI-generated responses.
- Familiarity with LLM evaluation techniques, conversational testing, and regression testing for AI applications.
- Experience monitoring chatbot quality in production environments.
- Understanding of test data management and quality metrics for AI-enabled systems.

Required Skills

Cloud Developer Cyber Vulnerability Management Analyst DevOps Engineer Fullstack Developer with GCP Mobile Architect Net Back Engineer NOC Network Engineer Office 365 Developer Web Development & Analytics Consultant

📌 Quality Engineer / SDET – AI Chatbot Validation (Toronto)
🏢 Realign
📍 Toronto