Member of Technical Staff - AI Infrastructure
About Us
Veeda AI is building the next generation of multimodal foundation world models for Physical AI. We're a small, fast-moving team of engineers and researchers from leading AI labs, tackling some of the most challenging problems at the intersection of AI, robotics, and embodied intelligence. If you're excited about pushing the boundaries of what's possible with Physical AI, you'll have the opportunity to make an outsized impact from day one.

Responsibilities

GPU Cluster Operations: Design, deploy, and operate bare-metal GPU clusters where Slurm and Kubernetes share nodes (Slinky, KubeRay, MPI Operator).

Scheduler & Topology: Tune Slurm block and topology plugins so a job lands inside one NVLink domain, and set fair-share, QoS, and preemption policy so long training runs and bursty simulation rollouts coexist.

Fabric Engineering: Own the interconnect (InfiniBand subnet manager, adaptive routing and SHARP, or RoCEv2 with PFC and ECN tuning), and validate it with nccl-tests before a hang gets blamed on the model.

Storage & Data Path: Run high-throughput storage (Lustre, WEKA, Ceph) with NVMe scratch tiers and caching so video datasets stream at line rate and checkpoint writes never stall a run.

Observability & Hardware Health: Build the telemetry pipeline (Prometheus, Grafana, DCGM, BMC/Redfish) that catches Xid and ECC errors, thermal throttling, and link flaps, and automate the drain-and-replace that keeps them off live jobs.

Requirements

Bachelor's degree in Computer Science, Computer Engineering, or equivalent hands‑on experience in high-performance computing (HPC) or infrastructure engineering.





Deep hands‑on expertise administering Linux-based HPC clusters running Slurm or Kubernetes (job scheduling, cgroups, fair-share policies, topology configuration, GPU device plugins).

Robust troubleshooting skills below the framework layer: low-level Linux networking, kernel and PCIe/NUMA tuning, hardware diagnostics, and network or distributed storage (NFS, NVMe-oF, Lustre, Ceph).

Proficient in automation and infrastructure-as-code tools (e.g., Ansible, Terraform, Helm, Python/Bash scripting), and treats cluster configuration as reviewed, version-controlled code.

Experience supporting distributed deep learning workloads (PyTorch/NCCL, Ray, DeepSpeed) closely enough to tell an infrastructure fault from a model bug, and able to prove which with a controlled benchmark.

Nice to Have

Experience managing high‑density GPU infrastructure (NVIDIA H100/H200, B200, and GB200 NVL72 systems, DGX/HGX architectures, liquid‑cooled racks).

Built container and image supply chains for HPC workloads (Enroot, Pyxis, Docker, custom Kubernetes operators).

Experience running hybrid capacity, combining owned hardware with cloud or neocloud burst under a single scheduler.

Experience running secure multi-tenant research environments with SSO, per-team quota, and interactive access that stays fast.

Contributed to Slurm plugins, Kubernetes operators, or open-source cluster and observability tooling.

Experience with GitLab, especially GitLab CI, for managing infrastructure-as-code and automation pipelines.

Experience benchmarking fabric, storage, or scheduler performance and publishing results internally or externally to settle a design or procurement decision.

#J-18808-Ljbffr

📌 Member of Technical Staff - AI Infrastructure (Ontario)
🏢 Veeda AI
📍 Ontario