Evaluate autonomous AI agents through rubric writing, trace debugging, and stress testing while working remotely as a freelance contributor for Outlier.

Outlier helps innovative companies improve AI agents by collecting human feedback. This work focuses on training large language models to function as proactive, multi-step systems. You will assess how agents design, coordinate, and optimize complex workflows. Applicants need at least two years of relevant experience.

Key Responsibilities:

- Debug agent traces across multi-turn interactions
- Write detailed evaluation rubrics
- Stress test autonomous agent behavior
- Assess complex system behaviors with precision
- Support training for generative AI systems

Requirements:

- Experience with backend engineering, AI automation, or systems integration
- Ability to maintain production-grade modular software
- Practical experience with live environments
- Explicit, high-density technical feedback skills
- Experience handling persistent state or session discovery preferred

Use technical feedback, agent testing, and systems integration experience to strengthen autonomous AI evaluation projects with Outlier.

📌 AI Agent Evaluation Engineer (Montreal)
🏢 Outlier
📍 Montreal