Distributed LLM performance engineering powers Huawei Canada’s Ascend NPU training systems with MindSpeed, hybrid parallelism, and profiling expertise. This Senior System Engineer position focuses on foundation-model pre-training and post-training at scale. You will architect distributed pipelines using MindSpeed, Megatron, PyTorch FSDP, veRL, vLLM, and Ray. The work targets MFU, compute-communication overlap, rollout pipelining, and reliable numerical behavior. Key Responsibilities:
Scale pre-training and RL post-training pipelines across NPU clusters Optimize hybrid TP, PP, CP, EP, and ZeRO parallelism Tune RL rollout and actor pipelining for training efficiency Implement FP8, MXFP8, and mixed-precision training schemes Eliminate pipeline bubbles, stalls,
and HCCL communication overhead Requirements:
Three-plus years in HPC, accelerator operators, or distributed training Practical distributed training experience with listed frameworks Proficiency in C/C++ and up-to-date Python systems programming Hands-on PyTorch custom C++ or CANN extension development Performance profiling experience on AI hardware Build scalable Ascend NPU training systems by combining distributed frameworks, parallelism tuning, profiling, and precision alignment.