19 Nov
|
Equinix
|
Toronto
Who are we?Equinix is the world's digital infrastructure company®, shortening the path to connectivity to enable the innovations that enrich our work, life and planet.A place where tech thinkers and future builders turn bold ideas into breakthrough experiences, we welcome your unique perspective. Help us challenge assumptions, uncover bias, and remove barriers—because progress starts with fresh ideas. You'll find belonging, purpose, and a team that welcomes you—because when you feel valued, you're empowered to do your best work.Job SummaryDesigns, develops and deploys the evaluation and trust capabilities of Equinix's GenAI platform. Builds platform features that let teams evaluate their agents at scale: golden datasets, synthetic test data, scoring with LLM-as-judge, and monitoring of agent quality in production. Builds and tests the trust controls that agents run on, including agent identity, access, guardrails and data boundaries. Traces agent failures to their root cause and fixes them in the platform. Works in a team with senior engineers and with product, platform and security partners to turn defined problems into robust, secure, observable and performant solutions.Knowledge and ApplicationApplies solid knowledge of software engineering practice and of LLM and agent concepts (prompting, retrieval, tool and function calling, orchestration, MCP). Understands evaluation methods (code-based checks, LLM-as-judge, human review, golden datasets, regression suites) and evaluation statistics (sampling, variance across runs, agreement between reviewers). Understands AI security risks such as prompt injection and data leakage, and identity concepts such as OAuth and per-user access.
Follows established standards and suggests improvements.Complexity and Problem SolvingWorks on problems of defined scope where analysis requires evaluation of identifiable factors. Traces an agent failure to its cause (retrieval, tool call, prompt, model, data or infrastructure) using logs and traces, and follows it until it is fixed. Judges whether a metric reflects real quality. Escalates unusual or high-impact issues. Work is reviewed at defined checkpoints.Collaboration and InteractionWorks with AI engineers, product managers, forward-deployed engineers, platform and security teams. Turns the needs of agent builders into platform features. Explains results, defects and risks to partners who are not familiar with the subject. Asks for guidance early and shares knowledge with the team.ResponsibilitiesEvaluation platform (primary focus)Build platform features for evaluation: dataset creation and versioning, golden datasets, synthetic test data and reusable metric templatesBuild scoring for agents: code-based checks, LLM-as-judge, tool-call accuracy and grounding checksBuild human-review features that compare judge scores against human reviewers, and recalibrate when agreement is lowRun evaluations at scale: before release, on every change to a prompt, model or tool, and on agents in productionBuild evaluation playbooks so agent builders can evaluate their agents without specialist helpKeep results traceable to the agent, model, prompt and dataset versions tested,
with a record of who approved each releaseTrust and safetyBuild and test the platform's trust controls: agent identity, per-user access to connected systems, tool permissions, guardrails and data boundariesRun adversarial tests and red-teaming (prompt injection, out-of-scope requests, data-boundary tests), and turn findings into fixes and regression testsWork with security and governance teams to turn policy requirements into automated checksRoot-cause analysis and qualityInvestigate agent failures and performance problems end to end, and follow them until the cause is fixedBuild test and performance tooling: end-to-end agent tests, load and latency tests, and failure diagnosticsTurn production failures and user feedback into regression tests, so the same defect does not returnQuality and responsible practiceWrite tested, reviewed and documented code, and contribute to shared libraries and toolingTest and build to security, data-privacy and responsible-AI requirementsQualificationsRequired1–4 years in AI/ML or software engineering (internships and applied research count)Bachelor's degree in computer science, engineering, data science, mathematics or a related field, or equivalent experienceProficiency in Python, with solid habits: version control, testing, code review. Experience building backend services or platform featuresHands-on experience building, not only using, evaluations or test systems for ML or LLM applications, including reading failures from logs or tracesWorking knowledge of LLM applications (prompting, retrieval, agents)Working knowledge of evaluation statistics (sampling, confidence intervals, variance across runs) and of measuring agreement between reviewersAbility to write SQL and analyze resultsClear written and verbal communication
📌 Ai Engineer, Trust & Evaluation (Toronto)
🏢 Equinix
📍 Toronto