Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.

As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable autonomous execution. You will curate Golden Sets, automate evaluation pipelines, audit reasoning trajectories, and tune prompts. Your work will also address tool calling, RAG pipelines, hallucinations, and guardrails. Collaboration with AI Solution Leads and AI Agent Developers supports robust agent components.

Key Responsibilities:

- Curate documents, queries, and expected agent trajectories
- Automate evaluations for accuracy, latency, token spend, and reliability
- Audit multi-step reasoning and identify logic deviations




- Optimize prompts, context windows, and few-shot examples
- Monitor drift, injections, loops, and hallucinations

Requirements:

- Three or more years in quality, automation, data, or ML engineering
- Recent LLM testing, prompt tuning, or orchestration experience
- Hands-on LangGraph, ADKs, or internal SDK experience
- JSON schema and function-calling expertise
- Python-heavy automation and asynchronous debugging experience

Apply analytical evaluation, Python automation, and LLM orchestration expertise to strengthen Equifax autonomous AI reliability.

📌 Agentic AI Evaluation Engineer (Toronto)
🏢 Equifax
📍 Toronto