23 Oct
|
Devopsroles
|
Toronto
23 Oct
Devopsroles
Toronto
Appnovation is the AI Product Company. Headquartered in Vancouver, Canada, with teams around the world, Appnovation builds AI products, delivers enterprise AI and digital solutions, and co-invests with clients to bring recent AI products to market. Open source has been the foundation of Appnovation's work since 2007, and it remains the foundation underneath every AI product the company builds today.
Appnovation teams work with organizations in highly regulated industries, including global leaders such as Pfizer, Google and AstraZeneca, to help them adopt AI they can govern and own. We’re looking for a Site Reliability Engineer to keep a shared observability platform for LLM-based applications running for a global life sciences client. The platform is built on Langfuse and self-hosted on Kubernetes on AWS, with ClickHouse as the analytical store, PostgreSQL for metadata and Redis as the ingestion queue, all delivered through Argo CD.
Two things need building rather than maintaining. The platform has no monitoring, alerting or defined service levels today, and you will own putting them in place.
Platform Operations: Diagnose and resolve failures across ClickHouse, PostgreSQL, Redis, the ingestion workers and the Kubernetes layer beneath them, including ingest backpressure from queue depth and worker drain behaviour. Rehearse the restore, time it, document it and test it against its failure modes, rather than assuming a successful backup job means a recoverable system.
Monitoring and Service Levels: Build monitoring, alerting and service levels from scratch,
so problems are found here before a user reports them.
Infrastructure as Code: Maintain Kubernetes manifests, Helm values and Argo CD applications so every change goes through the delivery pipeline.
Automation: Turn recurring operational work into automation.
Upgrade Partnership: Work with the platform engineer who owns what the platform offers. Hands-on experience with Kubernetes on AWS (managed EKS), with routine work done through Helm values and Argo CD applications.
Experience building monitoring and alerting from scratch, including service levels that reflect what users actually experience rather than what is easy to measure.
Experience upgrading self-hosted software safely, including schema migrations, rehearsal in a lower environment and a rollback plan for a partly completed migration. PostgreSQL and Redis operations deep enough to debug metadata-store and queue problems, including backpressure and worker drain. Observability engineering, including OpenTelemetry Collector pipelines, alerting design and Grafana dashboards.
OIDC or enterprise SSO integration with a corporate identity provider. GitHub Actions for plan and apply pipelines with approval gates. Thank you for your interest in a career with Appnovation Technologies!
Diversity, Equity & Inclusion is not only something that we embrace - we celebrate it! Come join us at Appnovation, and learn more about how we stay true to our company values as we build better lives through better digital. Accommodations are available upon request throughout the recruitment process. #
📌 Site Reliability Engineer, AI Observability Appnovation Technologies · Toronto · Site Reliability Pa
🏢 Devopsroles
📍 Toronto