Observability Engineer | Kubernetes, Prometheus, Grafana, Thanos, Loki

Role Description: Observability Engineer

Enterprise Kubernetes Platform | Financial Services

ABOUT THE ROLE

We are seeking an experienced Observability Engineer to join our Enterprise Kubernetes Platform team at a leading financial services organization. You’ll own the complete observability stack across 50+ production Kubernetes clusters, providing metrics, logging, tracing, and alerting capabilities that ensure exceptional reliability and performance for mission-critical applications.

This role combines deep technical expertise in contemporary observability tools with emerging AI/ML capabilities to build intelligent monitoring solutions, predictive alerting, and self-healing infrastructure.

WHAT YOU’LL DO

OBSERVABILITY STACK OWNERSHIP

- Design, deploy, and maintain enterprise-scale observability infrastructure including Prometheus, Grafana, Thanos, Loki, and modern collection agents.
- Manage observability deployments using GitOps principles and infrastructure as code.




- Implement long-term metrics storage solutions with cloud object storage.
- Maintain and upgrade observability components across development, QA, UAT, production, and DR environments.
- Configure distributed observability architecture spanning multiple datacenters and cloud providers.

METRICS & MONITORING

- Design and implement Prometheus monitoring strategies for Kubernetes infrastructure and containerized applications.
- Create ServiceMonitors, PodMonitors for automated metrics collection.
- Develop rules for intelligent alerting with minimal false positives.
- Configure multi-cluster metrics federation and aggregation.
- Optimize metrics cardinality, storage efficiency, and query performance.
- Implement recording rules for pre-aggregated metrics and SLI calculations.

DASHBOARDS & VISUALIZATION

- Build comprehensive Grafana dashboards for infrastructure health, application performance, and business metrics.
- Create reusable dashboard

📌 Observability Engineer (Halifax Regional Municipality)
🏢 Astra North Infoteck
📍 Halifax Regional Municipality