22 Nov
|
Equifax
|
Toronto
Synopsis of the roleAt Equifax, we are moving past passive AI chat interfaces to build the future of autonomous workflows. We are creating intelligent, self-correcting multi-agent systems that can navigate complex software environments, utilize external tools, and solve open-ended business problems with minimal human intervention. To ensure these systems are safe, reliable, and enterprise-grade, we are seeking an analytical Agentic AI Evaluation & Tuning Engineer.What you will doGolden Dataset Curation & Automated EvaluationBuild the 'Golden Set': Curate, maintain, and augment high-quality reference datasets (Golden Sets) of documents, user queries, and expected agent trajectories to serve as the ultimate source of truth for testing.Automate Eval Cycles: Design and implement automated, continuous evaluation pipelines to measure agent accuracy, latency, token spend, and fallback reliability before code hits production.Trajectory & Reasoning Auditing: Trace and dissect complex, multi-step agent 'thought' processes (e.G., ReAct, Reflection loops) to pinpoint exactly where an agent deviates from its intended logic path.Agent Tuning & Developer CollaborationBehavioral Optimization: Refine system prompts, context windows, and few-shot examples to optimize how agents execute complex, multi-step workflows.Tool & Function-Calling Optimization: Fine-tune how agents interact with external APIs, databases, and UiPath RPA workflows - minimizing execution errors, redundant calls, and token overhead.Augment Development: Partner closely with AI Solution Leads and AI Agent Developers to feed evaluation insights back into the development lifecycle, helping them build robust, reusable, and self-correcting agent components.Production Guardrails & Lifecycle Management (LLMOps)Defeat Drift & Hallucinations:
Actively monitor deployed agents to identify, troubleshoot, and mitigate semantic drift, prompt injections, infinite execution loops, and hallucinations.Maintain Autonomous Integrity: Implement robust guardrail frameworks to ensure agents maintain reliable, fact-based autonomous decision-making post-deployment in production.RAG & Knowledge Integration: Optimize Domain-Specific Knowledge Bases and Retrieval-Augmented Generation (RAG) pipelines to ensure agents pull from accurate data rather than assumptions.What Experience You NeedExperience: 3+ years of professional experience in software quality engineering, test automation, or data/ML engineering, with a dedicated focus on LLM testing, prompt tuning, or orchestration patterns over the last 1–2 years.Agentic & LLM Frameworks: Proven hands-on experience working with LLM orchestration frameworks (e.G., LangGraph, ADKs or specialized internal SDKs).Function Calling Mastery: Deep understanding of JSON schema design for LLM tool-calling, function-calling, and structured outputs.Advanced Debugging & Automation: Strong background in writing automated test scripts (Python-heavy) and using tracing/observability concepts to debug cascading errors in asynchronous, non-deterministic systems.What Could Set You ApartExperience with AI evaluation and observability platformsLive production experience testing Agentic workflows and GenAI solutionsFamiliarity with Google Cloud AI suite (Vertex & Gemini Enterprise Agent Platform) and UiPath ecosystem (Maestro).Experience utilizing LLMs to securely generate high-quality synthetic data for edge-case testing.Proficiency in Python or TypeScript, with a deep understanding of asynchronous programming, API design, and microservices architecture.Demonstrated learning agility and a proactive approach to mastering recent technologies.This is a newly created position.
📌 Agentic Ai Optimization Developer (Toronto)
🏢 Equifax
📍 Toronto