We are seeking an experienced Observability Engineer to join our Enterprise Kubernetes Platform team at a leading financial services organization. In this role, you will own the complete observability stack across 50 production Kubernetes clusters, delivering robust metrics, logging, tracing, and alerting capabilities to ensure exceptional reliability and performance for mission-critical applications.

This position bridges deep technical expertise in up-to-date cloud-native observability tools with emerging AI/ML capabilities to build intelligent monitoring solutions, predictive alerting systems, and self-healing infrastructure.

What You'll Do

- Observability Stack Ownership

- Architecture & Deployment: Design, deploy, and maintain enterprise-scale observability infrastructure, including Prometheus, Grafana, Thanos, Loki, and modern collection agents.
- GitOps & IaC: Manage all observability deployments using GitOps principles and Infrastructure as Code (IaC).
- Storage & Continuity: Implement long-term metrics storage solutions utilizing cloud object storage; maintain and upgrade components across Development, QA, UAT, Production, and DR environments.
- Multi-Cloud/Hybrid Setup: Configure distributed observability architectures spanning multiple data centers and cloud providers.

2. Metrics & Monitoring
- Strategy & Collection:



Design and implement Prometheus monitoring strategies for Kubernetes infrastructure and containerized workloads using ServiceMonitors and PodMonitors.
- Alerting Strategy: Develop rules for intelligent alerting designed to minimize noise and false positives.
- Federation & Aggregation: Configure multi-cluster metrics federation and aggregation across clusters.
- Performance Optimization: Optimize metrics cardinality, storage efficiency, query performance, recording rules, and SLI/SLO calculations.

3. Dashboards & Visualization
- Grafana Management: Build comprehensive Grafana dashboards for infrastructure health, application performance, and business metrics.
- Dashboard-as-Code: Create reusable dashboard templates and libraries for internal development teams using code-driven approaches.
- Governance & Access: Configure multi-datasource setups, role-based access control (RBAC), and multi-tenancy in Grafana.
- Executive Metrics: Design executive-level dashboards with clear SLO/SLI tracking and business KPIs.

4. Logging Infrastructure
- Centralized Logging: Deploy and manage centralized logging solutions (Loki, ELK, Splunk, or similar).
- Log Collection Agents: Configure log collection agents (Promtail, Fluentd, Fluent Bit, Vector, etc.).
- Cost & Compliance: Design log retention policies that effectively balance operational needs, compliance requirements, and cloud costs.
- Log Analytics: Write and optimize LogQL/Lucene queries to extract actionable log-based insights.

📌 Observability Engineer (Halifax Regional Municipality)
🏢 Envision Technology Solutions
📍 Halifax Regional Municipality