Shape LLM platform observability at BMO, using OpenTelemetry, Prometheus, and Grafana in a hybrid engineering setting.

The Senior AI Engineer builds capabilities for tracing agent workflows and managing production AI behavior. Responsibilities include capturing spans, prompts, tool calls, and intermediate outputs, while controlling token costs and log cardinality. You will also standardize pipelines, configure dashboards, and support automated anomaly detection and incident response.

Key Responsibilities:

- Implement OpenTelemetry-based agent tracing frameworks
- Capture prompts, spans, tools, and outputs
- Build telemetry strategies for token optimization
- Configure logging sidecars and centralized dashboards
- Replace static thresholds with anomaly detection

Requirements:

- Bachelor's degree in Computer Science or Engineering
- Seven to ten years of Site Reliability Engineering experience
- Experience automating detection, investigation, and remediation
- Python experience for custom evaluations
- SQL or LogQL/PromQL querying experience

Use LLM tracing, platform engineering, and intelligent monitoring to govern BMO’s autonomous AI workflows.

📌 Senior LLM Platform Observability Engineer (Toronto)
🏢 Aimlroles
📍 Toronto