Evaluate autonomous agents through rubric writing, trace debugging, and stress testing. Work remotely with Outlier as a freelance contributor on flexible schedules.

Outlier helps cutting-edge companies improve AI agents by providing human feedback for advanced generative systems. This work focuses on teaching large language models to coordinate multi-step, real-world architectural workflows. You will assess complex system behavior and deliver high-density technical feedback. The opportunity requires two or more years in backend engineering, AI automation, or complex systems integration. Experience with modular production software, live environments, and multi-turn interactions is required. Applicants may apply from Australia, Canada, Finland, France, Italy, Norway, the United Kingdom, or the United States.

Key Responsibilities:





- Develop evaluation rubrics for agent performance
- Debug traces from multi-step agent workflows
- Stress test autonomous agent behavior
- Identify privacy leaks and authority escalation
- Review indirect prompt injection failures

Requirements:

- At least two years of relevant experience
- Strong attention to complex system details
- Clear, high-density technical feedback skills
- Production-grade modular software experience
- Live, multi-turn systems experience

Use evaluation, debugging, and systems-integration expertise to strengthen autonomous-agent performance for Outlier’s AI projects.

📌 Autonomous Agent Evaluation Engineer (Victoria)
🏢 Outlier
📍 Victoria