Role Name: Sr. ML Systems Engineer, LLM Inference Performance on AWS Trainium

Location: Canada (Remote)

Role overview

· We are seeking a hands-on machine learning systems engineer to optimize production-grade large language model inference across current and next-generation AWS Trainium platforms, including Trainium2 and Trainium3.

· The work focuses on decode latency, throughput under representative load, price-performance, long-context and multi-chip scaling, and reproducible benchmarking against leading GPU platforms.

· As part of a small, senior engineering team, you will own substantial workstreams end to end, from model-level optimization through measurement, analysis, and documentation. You will communicate technical findings and business implications clearly to customer engineers and leadership.

· This is a hands-on, customer-facing role focused on delivering measurable production outcomes.

Key responsibilities

· Lead decode optimization. Improve decode latency, throughput, and cost on AWS Trainium platforms, validating correctness at every step. Initial priorities include speculative decoding and native multi-token prediction (MTP) for Qwen-family models in vLLM on AWS Neuron.

· Benchmark and assess performance rigorously. Design and run matched, reproducible tests against leading GPU platforms, controlling prompts, concurrency, cache state, and warm-up. Deliver latency-throughput Pareto frontiers and price-performance comparisons aligned with customer service-level objectives.

· Lead cross-generation optimization. Benchmark and migrate workloads across Trainium generations, including Trainium2 and Trainium3. Scale serving on multi-chip Trainium systems for long-context workloads, including contexts of 100K tokens or more,



and apply architecture-specific optimizations.

· Extend model coverage. Onboard additional architectures, including hybrid state-space and mixture-of-experts models, reusing and generalizing existing implementations where practical.

· Enable customer ownership. Create runbooks, implementation guides, and benchmark documentation, and support hands-on workshops so customer engineers can operate, reproduce, and tune the stack independently.

What success looks like

· Decode latency and serving cost improve measurably while maintaining model quality.

· Performance results are reproducible, correctly scoped, and defensible across hardware platforms.

· Final benchmarks reflect representative customer workloads, not synthetic peak numbers alone.

· Customer engineers can reproduce the environment, run the benchmarks, and continue optimization independently.

Basic qualifications

· 5+ years of experience in machine learning or systems engineering, including 2+ years focused on production model inference or serving.

· Solid proficiency in Python and PyTorch, with the ability to read and modify model implementations involving attention, mixture-of-experts routing, and KV-cache management.

· Hands-on experience with an LLM serving framework such as vLLM, TensorRT-LLM, SGLang, or Triton Inference Server.

· Deep understanding of LLM inference performance, including prefill and decode,



continuous batching, and tensor and expert parallelism, and their impact on latency, throughput, and cost.

· Experience profiling and optimizing accelerator workloads at the framework, compiler, or kernel level, and designing rigorous performance experiments.

· Strong written and verbal communication skills, with the ability to explain engineering trade-offs to technical audiences and business impact to senior stakeholders.

Preferred qualifications

· Experience with AWS Trainium or Inferentia and the AWS Neuron SDK, including NxD Inference, vLLM on Neuron, or Neuron Kernel Interface (NKI), or with migrating workloads across accelerator generations.

· Experience implementing or evaluating speculative decoding, such as draft-model approaches, Medusa, EAGLE, or MTP.

· Kernel development experience in NKI, Triton, or CUDA.

· Experience with hybrid or state-space architectures such as Mamba, large mixture-of-experts models, or FP8 / INT8 quantization with model quality and accuracy validation.

· Experience in a customer-facing engineering, consulting, or technical enablement role.

How we work

· Reproducibility: Every reported result has a clearly stated workload, configuration, pricing basis, and measurement method.

· Correctness before optimization: We validate model quality and output correctness before accepting any performance gain.

· End-to-end ownership: Engineers move from implementation through measurement, analysis, documentation, and customer communication.

· Customer enablement: Deliverables are designed so the customer can reproduce and extend the work independently.

Regards

Krishna

Pay: $55.00-$60.00 per hour

Work Location: Remote

📌 Senior ML Systems Engineer, LLM Inference Performance on AWS Trainium (Canada)
🏢 United Softw
📍 Canada