Production Reliability Engineer, AI Platform (Toronto)

Production Reliability Engineer, AI Platform (Toronto)

17 Nov
|
Equinix
|
Toronto

17 Nov

Equinix

Toronto

Who are we?

Equinix is the world’s digital infrastructure company®, shortening the path to connectivity to enable the innovations that enrich our work, life and planet.

A place where tech thinkers and future builders turn bold ideas into breakthrough experiences, we welcome your unique perspective.Help us challenge assumptions, uncover bias, and remove barriers—because progress starts with fresh ideas. You’ll find belonging, purpose, and a team that welcomes you—because when you feel valued, you’re empowered to do your best work.

Job Summary

Keeps Equinix's GenAI products and platforms secure and reliable, and makes them more resilient over time. Writes the code and automation that prevents failures, rather than only responding to them. Defines service-level objectives, builds monitoring and alerting, hardens services against dependency and capacity failures, and tests failure and recovery before they happen in production. Leads incident response when issues occur and turns each incident into a lasting fix. Works in a team with senior engineers and with product, platform and security partners to keep AI products, agents, connectors and self-hosted models robust, secure, observable and performant.

Knowledge and Application

Applies solid knowledge of software engineering and site reliability engineering (SRE) practice: service-level objectives (SLOs), error budgets, monitoring, alerting, incident response, CI/CD and infrastructure as code. Understands how distributed systems fail and how to contain those failures (timeouts, retries, circuit breakers, graceful degradation). Understands how GenAI products behave in production, including model latency, rate limits, quotas, streaming and token cost.



Follows established standards and suggests improvements.

Complexity and Problem Solving

Works on problems of defined scope where analysis requires evaluation of identifiable factors. Diagnoses failures across application, model, data, pipeline and infrastructure layers, and fixes the underlying cause rather than the symptom. Applies existing runbooks and approaches, and improves them. Escalates unusual or high-impact issues. Work is reviewed at defined checkpoints.

Collaboration and Interaction

Works with AI engineers, product managers, platform and security teams. Reviews designs and releases for reliability risk before they ship. Explains status, incidents and results clearly to partners who are not familiar with the subject. Asks for guidance early and shares knowledge with the team.

Responsibilities

Reliability and resilience engineering (primary focus)

- Write code and automation that makes GenAI products and platforms more resilient: retries, timeouts, circuit breakers, rate limiting, graceful degradation and fallback between models and providers
- Run load tests, failure-injection tests and game days to find weak points before users do, and fix what they reveal
- Harden services running on Kubernetes: health checks, autoscaling, disruption budgets, resource limits and multi-zone deployment




- Plan capacity ahead of launches and growth, including model-provider quotas, GPU and inference capacity, and database headroom
- Maintain backup, restore and disaster-recovery plans, and test recovery on a regular schedule
- Automate repeated operational tasks, and reduce manual work each quarter

Monitoring and service levels

- Define and track service-level indicators and objectives with product owners: availability, latency, error rate, token cost and quality scores
- Instrument services with logs, metrics and traces (e.g., OpenTelemetry), from the user request through agent, tool and model calls
- Build and tune dashboards and alerts for user-facing failures, and remove alerts that are noisy or not actionable

Release and deployment

- Support release readiness for GenAI products and platforms: checks for monitoring, alerts, runbooks, capacity, security review and rollback plan
- Deploy through CI/CD and GitOps pipelines, using containers and infrastructure as code, with canary rollouts and fast rollback

Incident response

- Take part in the on-call rotation, respond to incidents, and write blameless post-incident reviews
- Track post-incident actions to completion, so the same failure does not happen twice
- Write and update runbooks, and turn recurring incidents into automation or fixes

GenAI and self-hosted models

- Support the reliability of self-hosted open-weight and small language models: serving, GPU usage, scaling, rollout and rollback
- Monitor model and provider latency, errors, rate limits and cost, and flag capacity and cost risks
- Apply security, patching, secrets and data-boundary requirements to every deploym

📌 Production Reliability Engineer, AI Platform (Toronto)
🏢 Equinix
📍 Toronto

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: production reliability engineer, ai platform (toronto) / toronto

Subscribe to this job alert:

Get the latest job offers by email for: production reliability engineer, ai platform (toronto) / toronto