Improve AI agent reliability at TribalScale with evaluation pipelines, quality metrics, and disciplined testing for agentic systems. The location remains to be confirmed.This Agentic QA Engineer role focuses on validating AI-driven workflows in real-world environments. You will design evaluation datasets, test scenarios, and benchmarks for dynamic application outputs. You will also collaborate with AI engineers, developers, product teams, and client-facing engineering teams.Key Responsibilities:

Develop pipelines measuring agent accuracy and task completionEstablish quality metrics, datasets, scenarios, and benchmarksEvaluate edge cases, recovery, security, and resilienceIntegrate quality checks into CI/CD pipelinesInvestigate production issues and improve test coverageRequirements:

Robust analytical thinking for complex real-world test casesExperience creating automated tests and testing frameworksStrong API, integration, regression, and end-to-end testing knowledgeExperience debugging applications and technical issuesFamiliarity with modern software development practicesBring quality engineering expertise to AI agent evaluation, production reliability, and measurable software performance at TribalScale.

📌 Ai Agent Reliability Engineer (Winnipeg)
🏢 TribalScale
📍 Winnipeg