21 Nov
|
Equifax
|
Ontario
Synopsis of the role
At Equifax, we are moving past passive AI chat interfaces to build the future of autonomous workflows. We are creating intelligent, self-correcting multi-agent systems that can navigate complex software environments, utilize external tools, and solve open-ended business problems with minimal human intervention. To ensure these systems are secure, reliable, and enterprise-grade, we are seeking an analytical Agentic AI Evaluation & Tuning Engineer.
What you will do
Golden Dataset Curation & Automated Evaluation
Build the 'Golden Set': Curate, maintain, and augment high-quality reference datasets (Golden Sets) of documents, user queries, and expected agent trajectories to serve as the ultimate source of truth for testing.
Automate Eval Cycles: Design and implement automated, continuous evaluation pipelines to measure agent accuracy, latency, token spend, and fallback reliability before code hits production.
Trajectory & Reasoning Auditing: Trace and dissect complex, multi-step agent 'thought' processes (e.g., ReAct, Reflection loops) to pinpoint exactly where an agent deviates from its intended logic path.
Agent Tuning & Developer Collaboration
Behavioral Optimization: Refine system prompts, context windows, and few-shot examples to optimize how agents execute complex, multi-step workflows.
Tool & Function-Calling Optimization: Fine-tune how agents interact with external APIs, databases, and UiPath RPA workflows - minimizing execution errors, redundant calls, and token overhead.
Augment Development: Partner closely with AI Solution Leads and AI Agent Developers to feed evaluation insights back into the development lifecycle, helping them build robust, reusable, and self-correcting agent components.
Production Guardrails & Lifecycle Management (LLMOps)
Defeat Drift & Hallucinations: Actively monitor deployed agents to identify,
troubleshoot, and mitigate semantic drift, prompt injections, infinite execution loops, and hallucinations.
Maintain Autonomous Integrity: Implement robust guardrail frameworks to ensure agents maintain reliable, fact-based autonomous decision-making post-deployment in production.
RAG & Knowledge Integration: Optimize Domain-Specific Knowledge Bases and Retrieval-Augmented Generation (RAG) pipelines to ensure agents pull from accurate data rather than assumptions.
What Experience You Need
Experience: 3+ years of professional experience in software quality engineering, test automation, or data/ML engineering, with a dedicated focus on LLM testing, prompt tuning, or orchestration patterns over the last 1–2 years.
Agentic & LLM Frameworks: Proven hands-on experience working with LLM orchestration frameworks (e.g., LangGraph, ADKs or specialized internal SDKs).
Function Calling Mastery: Deep understanding of JSON schema design for LLM tool-calling, function-calling, and structured outputs.
Advanced Debugging & Automation: Strong background in writing automated test scripts (Python-heavy) and using tracing/observability concepts to debug cascading errors in asynchronous, non-deterministic systems.
What Could Set You Apart
Experience with AI evaluation and observability platforms
Live production experience testing Agentic workflows and GenAI solutions
Familiarity with Google Cloud AI suite (Vertex & Gemini Enterprise Agent Platform) and UiPath ecosystem (Maestro).
Experience utilizing LLMs to securely generate high-quality synthetic data for edge-case testing.
Proficiency in Python or TypeScript, with a deep understanding of asynchronous programming, API design, and microservices architecture.
Demonstrated learning agility and a proactive approach to mastering new technologies.
This is a newly created position.
#J-18808-Ljbffr
📌 Agentic AI Optimization Developer (Ontario)
🏢 Equifax
📍 Ontario