Distributed LLM performance engineering powers Huawei Canada’s Ascend NPU training systems with MindSpeed, hybrid parallelism, and profiling expertise.
This Senior System Engineer position focuses on foundation-model pre-training and post-training at scale. You will architect distributed pipelines using MindSpeed, Megatron, PyTorch FSDP, veRL, vLLM, and Ray. The work targets MFU, compute-communication overlap, rollout pipelining, and reliable numerical behavior.
Key Responsibilities:
- Scale pre-training and RL post-training pipelines across NPU clusters
- Optimize hybrid TP, PP, CP, EP, and ZeRO parallelism
- Tune RL rollout and actor pipelining for training efficiency
- Implement FP8, MXFP8, and mixed-precision training schemes
- Eliminate pipeline bubbles,
stalls, and HCCL communication overhead
Requirements:
- Three-plus years in HPC, accelerator operators, or distributed training
- Practical distributed training experience with listed frameworks
- Proficiency in C/C++ and up-to-date Python systems programming
- Hands-on PyTorch custom C++ or CANN extension development
- Performance profiling experience on AI hardware
Build scalable Ascend NPU training systems by combining distributed frameworks, parallelism tuning, profiling, and precision alignment.
📌 Senior Distributed LLM Performance Engineer (British Columbia)
🏢 Huawei Canada
📍 British Columbia
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.