Improve AI agent reliability at TribalScale with evaluation pipelines, quality metrics, and disciplined testing for agentic systems. The location remains to be confirmed.

This Agentic QA Engineer role focuses on validating AI-driven workflows in real-world environments. You will design evaluation datasets, test scenarios, and benchmarks for dynamic application outputs. You will also collaborate with AI engineers, developers, product teams, and client-facing engineering teams.

Key Responsibilities

Develop pipelines measuring agent accuracy and task completion

Establish quality metrics, datasets, scenarios, and benchmarks

Evaluate edge cases, recovery, security, and resilience

Integrate quality checks into CI/CD pipelines





Investigate production issues and improve test coverage

Requirements

Strong analytical thinking for complex real-world test cases

Experience creating automated tests and testing frameworks

Strong API, integration, regression, and end-to-end testing knowledge

Experience debugging applications and technical issues

Familiarity with up-to-date software development practices

Bring quality engineering expertise to AI agent evaluation, production reliability, and measurable software performance at TribalScale.

📌 AI Agent Reliability Engineer (Winnipeg)
🏢 TribalScale
📍 Winnipeg