- AI/ML Platform Engineer – Containers & GPU Infrastructure (Sr Kubernetes & AI Platform Engineer)
- Location: Toronto, ON (Hybrid)
- Type: Contract-to-Hire

Position Summary:

- We are seeking a highly experienced Senior Kubernetes & AI Platform Engineer to design, implement, and operationalize secure, production-ready GPU capabilities across Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE) and Google Distributed Cloud (GDC). The role will establish standardized platform patterns for AI and machine learning workloads, with a strong focus on GPU lifecycle management, resource efficiency, observability, automation, security, and operational support.
- The successful candidate will work across cloud engineering, platform engineering, security, network, governance, and application teams. This is a hands-on engineering role requiring deep Kubernetes expertise, practical NVIDIA GPU experience, strong automation skills, and the ability to translate emerging AI infrastructure capabilities into governed enterprise services.

Key Accountabilities:

GPU and AI Platform Enablement:

- Design and implement production-grade GPU-enabled Kubernetes platforms on AKS, GKE or GDC.
- Deploy, configure, upgrade, and operate the NVIDIA GPU Operator and its supporting components.
- Establish a repeatable framework for NVIDIA driver installation, validation, lifecycle maintenance, rollback, and compatibility testing.
- Implement GPU discovery, health monitoring, telemetry, alerting, and capacity dashboards using NVIDIA DCGM Exporter integrated with enterprise observability platforms.
- Enable and govern GPU partitioning and sharing capabilities, including Multi-Instance GPU (MIG), time-slicing, and Dynamic Resource Allocation (DRA), where supported and appropriate.
- Define scheduling patterns for GPU workloads, including resource requests and limits, node selectors, affinities, taints and tolerations, topology constraints, priority, quotas, and workload isolation.
- Validate platform patterns for high-end accelerator types, including H100/H200-class GPUs and future supported accelerator families.
- Partner with AI/ML engineering teams to support model training, fine-tuning, inference, notebooks, batch processing, and distributed workloads.

Automation, Scalability, and Cost Optimization:

- Automate provisioning and configuration using Terraform, Helm, GitHub Actions, and cloud-native APIs or SDKs.
- Implement safe scale-up and scale-down patterns for system, compute, and user node pools, including scale-to-zero where platform and workload constraints permit.
- Develop controls for GPU quotas, reservations, utilization, idle-resource detection, chargeback/showback data, and capacity planning.
- Optimize cluster autoscaling, node auto-provisioning, workload placement, startup performance, and accelerator consumption.
- Evaluate and implement image streaming or equivalent acceleration techniques for large AI workload images and model-serving dependencies.
- Evaluate workload checkpointing and pod snapshot capabilities to improve recovery, portability, and efficient use of expensive GPU capacity.

Security, Compliance, and Supply Chain:





- Implement hardened Kubernetes and node configurations aligned with enterprise standards, STIG requirements, CIS guidance, and cloud-provider security controls.
- Integrate Wiz policies and findings into platform guardrails, deployment workflows, exception processes, and remediation practices.
- Apply least-privilege access controls using Kubernetes RBAC, cloud identity, workload identity, managed identities, and service accounts.
- Implement policy-as-code controls for privileged workloads, host access, Linux capabilities, device access, network policy, approved registries, signed images, and required security context settings.
- Perform threat modelling and security assessments for GPU operators, device plugins, drivers, workload runtimes, model-serving components, and supporting infrastructure.

Reliability, Observability, and Day 2 Operations:

- Define service-level indicators, alert thresholds, dashboards, health checks, and operational readiness criteria for GPU-enabled clusters.
- Troubleshoot Kubernetes scheduling, GPU allocation, driver, firmware, device plugin, container runtime, network, storage, and performance issues.
- Develop operational documentation, support procedures, troubleshooting guides, maintenance plans, and production runbooks.
- Define the Day 2 support model, escalation paths, service ownership boundaries, and RACI across platform, cloud, security, vendor, and application teams.
- Plan and execute upgrades, patching, disaster recovery tests, resilience exercises, incident response, and post-incident corrective actions.
- Create automated conformance and regression tests to validate GPU functionality, security controls, performance, and platform compatibility before release.

Engineering Leadership and Collaboration:

- Provide technical leadership for Kubernetes and AI infrastructure design decisions across AKS, GKE and GDC.
- Mentor application teams on secure, reliable, and cost-effective GPU workload patterns.
- Produce architecture documentation, technical standards, decision records, implementation guidance, and executive-ready recommendations.
- Manage engineering work through an agile backlog, including features, technical debt, defects, risk remediation, and platform enhancements.
- Collaborate with Microsoft, Google Cloud, NVIDIA, security partners, and internal infrastructure teams to resolve product and integration issues.

Key Deliverables:

- A standardized, production-ready reference architecture for GPU-enabled AKS, GKE and GDC clusters.
- Automated GPU Operator, driver, device plugin, and supporting component deployment and lifecycle management.
- DCGM-based GPU monitoring, dashboards, alerting, and operational integration.
- Validated DRA, MIG,



and GPU-sharing patterns with documented eligibility, constraints, and workload guidance.
- Automated node-pool scaling and scale-to-zero capabilities with appropriate safeguards.
- STIG-aligned baseline configurations and Wiz policy integration.
- Operational runbooks, support model, RACI, upgrade procedures, and incident response guidance.
- Implement image-streaming and workload checkpointing capabilities, where technically viable.

Required Qualifications and Experience:

- 7+ years of experience in cloud, platform, DevOps, SRE, or infrastructure engineering, including substantial hands-on Kubernetes experience.
- Demonstrated experience designing, deploying, and operating production Kubernetes environments on AKS, GKE or GDC.
- Hands-on experience with NVIDIA GPUs in Kubernetes, including GPU Operator, drivers, container toolkit, device plugins, DCGM/DCGM Exporter, MIG, and GPU scheduling.
- Strong understanding of Kubernetes architecture, controllers, scheduling, networking, storage, security, observability, upgrades, and failure modes.
- Experience with Kubernetes autoscaling, node-pool management, cluster autoscaler, workload placement, quotas, and resource optimization.
- Strong Infrastructure-as-Code and automation experience using Terraform, Helm, GitHub Actions, and scripting in Python, Go, or Bash.
- Experience implementing security and compliance controls using Kubernetes admission policies, cloud-native policy services, and security platforms such as Wiz or equivalent.
- Experience with Prometheus-compatible metrics, cloud monitoring services, centralized logging, alerting, and incident response.
- Solid knowledge of Linux, container runtimes, OCI images, registries, networking, storage, PKI, identity, and secrets management.
- Excellent written and verbal communication skills, with the ability to explain complex engineering decisions to technical and non-technical stakeholders.

Preferred Qualifications:

- Experience supporting AI/ML platforms, model training, inference, distributed computing, notebooks, or MLOps workflows.
- Experience with GPU performance profiling, capacity planning, topology awareness, RDMA, high-performance networking, or distributed AI frameworks.
- Knowledge of model-serving platforms and frameworks such as KServe, Kubernetes Gateway API, NVIDIA NIM, Triton Inference Server, Ray, Kubeflow, or MLflow.
- Experience with Azure Machine Learning and Google Vertex AI integrations with Kubernetes-based platforms.
- Experience with confidential computing, sandboxed runtimes, service mesh, policy-as-code, software supply-chain security, and regulated financial-services environments.
- Professional certifications such as CKA, CKS, CKAD, Microsoft Azure Administrator or DevOps Engineer, Google Skilled Cloud DevOps Engineer, or relevant NVIDIA certification.

Pay: $75.00-$90.00 per hour

Experience:

- AI/ML : 3 years (preferred)
- Platform Engineering: 10 years (preferred)
- Kubernetes: 10 years (preferred)
- NVIDIA GPUs in Kubernetes: 10 years (preferred)
- Terraform: 8 years (preferred)
- Python: 8 years (preferred)

Work Location: In person

📌 Sr Kubernetes & AI Platform Engineer (Toronto)
🏢 IT Accel
📍 Toronto