We are Bagel Labs

- a distributed machine learning research lab working towards open-source superintelligence.

We ignore years of experience and pedigree.
If you have high agency

- meaning your default assumption is that you can control the outcome of whatever situation you are in
- we want to hear from you.

Every requirement below is flexible for a candidate with high enough agency and tolerance for ambiguity.

Overview

You will design and optimize a distributed diffusion model training and serving system. Your focus is on building scalable, fault-tolerant infrastructure that can serve open-source diffusion models across multiple nodes and regions, with efficient support for adaptation techniques.

Key Responsibilities

- Design and implement distributed diffusion model inference systems for image, video, and multimodal generation across multiple nodes and regions.
- Architect high-availability clusters for diffusion model serving with automatic failover, load balancing, and dynamic batching for variable-resolution outputs.
- Build monitoring and observability systems for distributed diffusion inference (denoising steps, memory usage, generation latency, CLIP score tracking).
- Integrate with open-source diffusion frameworks (Diffusers, ComfyUI, Invoke AI) and optimize for production-scale serving.
- Implement and optimize cutting-edge techniques: rectified flow models, consistency distillation, and progressive distillation for few-step generation.
- Design distributed systems for Control Net, IP-Adapter, and multi-modal conditioning at scale.
- Build infrastructure for efficient LoRA/LyCORIS adaptation serving with hot-swapping and memory-efficient merging.




- Optimize VAE decoding pipelines and implement tiled/windowed generation for ultra-high-resolution outputs.
- Document architectural decisions, review code, and publish technical deep-dives on blog.bagel.com.

Who You Might Be

You have a deep understanding of distributed systems and diffusion model architectures. You/'re excited about the rapid evolution from DDPM to flow matching and consistency models. You enjoy architecting scalable infrastructure that can handle the unique challenges of diffusion models - from variable compute requirements per timestep to effective caching of intermediate states.

Desired Skills

- At least 5 years of experience with distributed systems and production ML serving.
- Hands-on experience with diffusion model frameworks (Diffusers, ComfyUI, or similar) in production environments.
- Deep understanding of diffusion model architectures (U-Net, DiT, rectified flows, consistency models).
- Experience with distributed GPU orchestration for high-memory workloads.
- Proven record of optimizing generation latency (classifier-free guidance, DDIM/DPM solvers, distillation techniques).
- Experience with attention optimization techniques (Flash Attention, xFormers, memory-efficient attention).
- Strong understanding of adaptation techniques (LoRA, LyCORIS, textual inversion, Dream Booth).
- Expertise in handling variable-resolution generation and dynamic batching strategies.

What We Offer

- Top of the market compensation.
- A deeply technical culture where bold, frontier ideas are debated, stress-tested, and built.
- Full remote flexibility within North American time zones.
- Ownership of work that can set the direction for decentralized AI.
- Paid travel opportunities to the top ML conferences around the world.

Please apply via our careers page. Note: we do not share application links here.

#J-18808-Ljbffr

📌 Machine Learning Engineer - Diffusion (Edmonton)
🏢 Bagel Labs
📍 Edmonton