Overview

We're looking for a solid QA Analyst to lead testing efforts for Agentic AI/LLM-based systems within a banking environment. You'll design and execute test strategies for autonomous agent workflows, ensuring accuracy, safety, and compliance in a highly regulated setting.

Key Responsibilities
Design test plans/cases for agentic AI systems (multi-step reasoning, tool-calling, orchestration workflows)
Validate LLM outputs for accuracy, hallucination risk, bias, and prompt-injection resilience
Perform functional, regression, integration, and UAT testing across agent pipelines and APIs
Build automated test frameworks (Python/PyTest/Selenium or similar) for agent behavior validation
Test human-in-the-loop and guardrail mechanisms for AI decision-making
Collaborate with data scientists, ML engineers, and product teams to define acceptance criteria
Document defects, edge cases, and evaluation metrics (precision/recall, response quality scoring)
Ensure compliance with bank's AI governance, risk, and regulatory standards (OSFI, model risk mgmt)

Required Qualifications
5+ years in QA/Test Analyst roles, ideally in financial services
Hands-on experience testing Agentic AI,



LLM applications, or RAG pipelines
Solid Python skills for test automation and scripting
Experience with prompt engineering evaluation, agent frameworks (LangChain, AutoGen, CrewAI, or similar)
Familiarity with API testing (Postman, REST/JSON validation)
Understanding of AI/ML testing challenges: non-determinism, drift, bias detection
Experience in Agile/Scrum environments
Banking/financial domain knowledge is a robust asset

Nice to Have
Experience with Databricks or cloud ML platforms
Exposure to AI governance/risk frameworks
Test automation for conversational AI/chatbots

Join us as a Senior QA Analyst where your expertise will elevate our quality standards! We’re committed to fostering an inclusive environment that encourages innovation through continuous learning and growth opportunities—empowering you to thrive professionally while contributing meaningfully to cutting-edge projects!

Pay: $65.00-$75.00 per hour

Expected hours: 40.0 per week

Work Location: Hybrid remote in Toronto, ON M5G 1Z5

📌 Senior Qa Analyst Toronto
🏢 Dyna Info Services
📍 Toronto