Build reliable autonomous AI at Equifax through LLM evaluation, Python automation, and multi-agent workflow testing. This newly created position focuses on production reliability.

As an Agentic AI Evaluation & Tuning Engineer, you will connect LLM capabilities with dependable autonomous execution. You will curate Golden Sets, automate evaluation pipelines, audit reasoning trajectories, and tune prompts. Your work will also address tool calling, RAG pipelines, hallucinations, and guardrails. Collaboration with AI Solution Leads and AI Agent Developers supports robust agent components.

Key Responsibilities

Curate documents, queries, and expected agent trajectories

Automate evaluations for accuracy, latency, token spend, and reliability

Audit multi-step reasoning and identify logic deviations





Optimize prompts, context windows, and few-shot examples

Monitor drift, injections, loops, and hallucinations

Requirements

Three or more years in quality, automation, data, or ML engineering

Recent LLM testing, prompt tuning, or orchestration experience

Hands-on LangGraph, ADKs, or internal SDK experience

JSON schema and function-calling expertise

Python-heavy automation and asynchronous debugging experience

Apply analytical evaluation, Python automation, and LLM orchestration expertise to strengthen Equifax autonomous AI reliability.

📌 Agentic AI Evaluation Engineer (Winnipeg)
🏢 Equifax
📍 Winnipeg