26 Oct
|
iFrame
|
Winnipeg
Own the physical reality of the platform in one of our seven regions. You bring up new GPU racks, validate Infini Band fabric end-to-end, and keep the cluster running at the SLA. This is a hands-on role; you will see the hardware.
Type Full-time
· IC4-IC6Stack Linux Infini Band (NDR / XDR) NCCL / RCCL Kubernetes (host-level) Terraform Prometheus / Grafana Go or PythonHiring manager replies within 5 business days.
The teamAbout the team
Cluster SRE is six engineers across the seven regions. Each region has a primary and a secondary; you will be one of those for your region. The team coordinates daily, deploys weekly, and rotates a global pager.
Reports to the head of cluster engineering. Primary on a single region; rotates secondary for one neighboring region.
What you'll do
01 Bring up recent B200 / B300 / MI300 X racks: cabling, To R config, NCCL/RCCL all-reduce validation, MFU baseline tests.02 Drive Infini Band fabric to spec
- NDR / XDR depending on the rack
- and chase residual bit-error budget down to zero.03 Run capacity planning across seven regions: forecast demand, model power and thermal headroom, work with procurement on lead times.04 Own the regional incident response. P1 incidents page within fifteen minutes; resolution target is four hours.05 Build and maintain the bring-up runbook so the second hire after you can do their first rack solo.06 Carry the global pager about one week per six, alongside runtime and customer engineering.
The barWhat we're looking for
Five-plus years operating large compute clusters — supercomputing centers, hyperscaler infra, or HPC at a national lab count.
Deep Infini Band and Ethernet Ro CE
experience:
subnet manager tuning, fabric debugging, lossless fort writing Go or Python for tooling. We are not strict about which.
Calm under load. You will be the named person on a $50 M-ARR
📌 Cluster site reliability engineer (Winnipeg)
🏢 iFrame
📍 Winnipeg