Client: City of Toronto (Technology Services)

Job Title: Systems Testing/QA Specialist – Senior (AI Testing)

Location: Hybrid: 3 Days onsite | Metro Hall, 55 John Street

Engagement: Contract Full-Time

Contract Duration: 130.00 Days (Potential Extension)

:

Must Have:

- Minimum of 3 years of solid experience in AI-assisted testing with hands-on experience in developing AI-powered QA agents using agentic AI, LLM technologies and chatbot applications to create and execute test cases.
- Minimum of 10+ years of experience performing testing for large-scale applications, including on-premise web solutions, cloud-based integrations, and mobile platforms.
- Extensive hands-on experience with all layers of testing: UI, Integration, Regression, Performance, AI-enabled systems, and User Acceptance Testing—across web-based, mobile, and cloud platforms.
- Proven ability to execute AI testing activities utilizing specialized tools such as DeepEval, Applitools, Promptfoo, SaolaAI, Reflect AI, Agentic AI, Copilot Studio Kit, etc.

NOTE: HM is looking for robust AI testing experience.

Public Sector Experience: Preferred

References are required

Responsibilities:

- The Quality Assurance Specialist (QA lead) is responsible for performing application testing tasks.
- The Quality Assurance activities including Functional Testing, Automation Testing, and various AI testing as described below.
- The Quality Assurance Specialist (QA lead) will participate in planning and estimating the activities required to test the new solutions and enhanced applications. He/She will participate in the development of Master Test Plans, Test
- Cases, Test Scripts, Test Completion Reports, etc. while providing advice to business users for the development of Test Cases for User Acceptance Test.
- Executing the tests as required, performing issue tracking and resolution.
- The Quality Assurance Specialist (QA Lead) plays a critical role in ensuring the delivery of high-quality software solutions by designing, executing, and managing testing activities throughout the software development lifecycle.
- The QA Specialist will work collaboratively with cross-functional teams to validate system functionality, performance, and usability, and ensure alignment with business requirements and quality standards.
- Quality Assurance Leadership
- Lead testing efforts for assigned projects, including planning, coordination,



execution and oversight of QA activities.
- Assess risks to quality based on verification and validation results and provide recommendations for mitigation.
- Implement and maintain quality processes that support development and production change management.
- Test Design & Planning
- Design comprehensive manual and automated testing processes, including test plans, test cases, and test scripts.
- Develop functional, system, regression, integration, load, and performance test scenarios based on project requirements.
- Collaborate with development leads and business analysts to design and create test data to support test execution.
- Test Execution & Management
- Perform various types of testing manually and/or using automation tools, including:
- Functional Testing
- Regression Testing
- Integration Testing
- Usability/Accessibility Testing
- Performance Testing
- Perform the following types of AI testing activities:
- Functional AI Testing
- Response accuracy
- Hallucination testing
- Prompt testing
- AI-Security and privacy testing (prompt injection)
- AI-Safety Testing (inappropriate behaviours)
- Execute test cycles, monitor progress, and manage defect resolution and issue tracking.
- Maintain detailed records of test results and discrepancies using designated QA tools.
- Coordination & Collaboration
- Work closely with project managers, implementation team, system administrators, architects, vendors (as needed) and database administrators to ensure testing activities are aligned with project timelines and technical requirements.
- Lead and assist business users in conducting User Acceptance Testing (UAT), providing guidance and support as needed.
- Reporting & Communication
- Manage the full-cycle process for documenting and communicating test discrepancies and defects to appropriate stakeholders.
- Provide regular updates on testing progress, issues, and outcomes to project teams and management
- Develop and build a Test Completion Report summarizing all testing activities and results for Gating purposes





Qualifications:

- Post-secondary degree in Computer Science, Electronic Engineering, or a related discipline.
- Minimum of 10+ years of experience performing testing for large-scale applications, including on-premise web solutions, cloud-based integrations, and mobile platforms.
- Minimum of 3 years of solid experience in AI-assisted testing with hands-on experience in developing AI-powered QA agents using agentic AI, LLM technologies and chatbot applications to create and execute test cases.
- Experience using AI tools to increase testing coverage, negative testing, regression testing with knowledge of AI-assisted automation for CI/CD pipelines.
- Experience leading and coordinating testing activities in collaboration with project with project teams and a strong understanding of the various delivery methodologies such as: SDLC - Waterfall, Agile (Scrum), Hybrid, and DevOps.
- Certification in Software Quality Engineering or an equivalent credential is considered a strong asset.
- Excellent analytical skills in business processes, information systems, information management, and web UI design and accessibility.
- Strong written and verbal communication skills, with the ability to collaborate effectively across multidisciplinary teams.

Technical Capabilities:

- Extensive hands-on experience with all layers of testing: UI, Integration, Regression, Performance, AI-enabled systems, and User Acceptance Testing—across web-based, mobile, and cloud platforms.
- Proven ability to execute AI testing activities utilizing specialized tools such as DeepEval, Applitools, Promptfoo, SaolaAI, Reflect AI, Agentic AI, Copilot Studio Kit, etc.
- Skilled in implementing automated testing using tools such as Micro Focus UFT, TOSCA, Selenium, Cucumber, as well as conducting performance testing with JMeter and OpenText Load Runner.
- As well as experience with platforms such as: ServiceNow, Salesforce, SAP, or equivalent are considered an asset
- Strong experience in executing test cycles, analyzing results, and managing the full defect lifecycle using tools like OpenText ALM, OpenText Octane, JIRA, Visual Studio, Confluence, etc.
- Demonstrated capability in analyzing business requirements and user stories to design test scenarios, develop test cases and scripts, prepare test data, and assess results for quality assurance.

📌 Senior QA Specialist / QA Lead (AI Testing Focus) (Toronto)
🏢 GlobalVariable
📍 Toronto