Build AI observability systems at BMO in a hybrid role, using Python, telemetry, and monitoring to evaluate production models and agents.This senior engineering role focuses on artificial intelligence observability and platform capabilities. You will trace LLM calls, monitor token usage, detect drift, and govern autonomous agent behaviors. The work also includes machine learning systems, experiments, algorithms, and specialized technical support.Key Responsibilities:
Design systems tracking, evaluating, and debugging non-deterministic AI modelsImplement OpenTelemetry tracing for agent trajectories, prompts, and tool callsBuild automated evaluations for relevance, groundedness, and hallucinationsMonitor drift, latency, errors, token usage,



and data qualityDefine guardrails and self-remediation boundaries for autonomous agentsRequirements:
Bachelor's degree in Computer Science, Engineering, or relevant domainSeven to ten years of Site Reliability Engineering experienceArtificial intelligence observability engineering and automation experiencePython plus SQL, LogQL, or PromQL experiencePrometheus, Grafana, or Datadog observability pipeline experienceApply senior AI engineering expertise across observability, evaluation, tracing, monitoring, and agent governance at BMO.
J-18808-Ljbffr

📌 Senior Ai Observability Engineer Winnipeg
🏢 Bank N.a.
📍 Winnipeg