Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable autonomous execution. You will curate Golden Sets, automate evaluation pipelines, audit reasoning trajectories, and tune prompts.

Your work will also address tool calling, RAG pipelines, hallucinations, and guardrails. Collaboration with AI Solution Leads and AI Agent Developers supports robust agent components.Key Responsibilities:

Curate documents, queries, and expected agent trajectoriesAutomate evaluations for accuracy, latency, token spend,



and reliabilityAudit multi-step reasoning and identify logic deviationsOptimize prompts, context windows, and few-shot examplesMonitor drift, injections, loops, and hallucinationsRequirements:

Three or more years in quality, automation, data, or ML engineeringRecent LLM testing, prompt tuning, or orchestration experienceHands-on LangGraph, ADKs, or internal SDK experienceJSON schema and function-calling expertisePython-heavy automation and asynchronous debugging experienceApply analytical evaluation, Python automation, and LLM orchestration expertise to strengthen Equifax autonomous AI reliability.

📌 Agentic Ai Evaluation Engineer (Winnipeg)
🏢 Equifax
📍 Winnipeg