Overview

We're looking for a strong QA Analyst to lead testing efforts for Agentic AI/LLM-based systems within a banking workplace. You'll design and execute test strategies for autonomous agent workflows, ensuring accuracy, safety, and compliance in a highly regulated setting.

*Key Responsibilities*

- Design test plans/cases for agentic AI systems (multi-step reasoning, tool-calling, orchestration workflows)

- Validate LLM outputs for accuracy, hallucination risk, bias, and prompt-injection resilience

- Perform functional, regression, integration, and UAT testing across agent pipelines and APIs

- Build automated test frameworks (Python/PyTest/Selenium or similar) for agent behavior validation

- Test human-in-the-loop and guardrail mechanisms for AI decision-making

- Collaborate with data scientists, ML engineers, and product teams to define acceptance criteria

- Document defects, edge cases, and evaluation metrics (precision/recall, response quality scoring)

- Ensure compliance with bank's AI governance, risk, and regulatory standards (OSFI, model risk mgmt)

*Required Qualifications*

- 5+ years in QA/Test Analyst roles, ideally in financial services

- Hands-on experience testing Agentic AI,



LLM applications, or RAG pipelines

- Strong Python skills for test automation and scripting

- Experience with prompt engineering evaluation, agent frameworks (LangChain, AutoGen, CrewAI, or similar)

- Familiarity with API testing (Postman, REST/JSON validation)

- Understanding of AI/ML testing challenges: non-determinism, drift, bias detection

- Experience in Agile/Scrum environments

- Banking/financial domain knowledge is a strong asset

*Nice to Have*

- Experience with Databricks or cloud ML platforms

- Exposure to AI governance/risk frameworks

- Test automation for conversational AI/chatbots

Join us as a Senior QA Analyst where your expertise will elevate our quality standards! We’re committed to fostering an inclusive environment that encourages innovation through continuous learning and growth opportunities—empowering you to thrive professionally while contributing meaningfully to cutting-edge projects!

Pay: $65.00-$75.00 per hour

Expected hours: 40.0 per week

Work Location: Hybrid remote in Toronto, ON M5G 1Z5

📌 Senior QA Analyst (Canada)
🏢 Dyna Info Services
📍 Canada