Sign in
← AI PulseSep 14, 2026

Wire · analysis · Single-source brief

NVIDIA Transformer Engine Accelerates Dropless MoE Training in JAX

NVIDIA Transformer Engine with JAX has delivered a 10.4x throughput improvement for Mixture of Experts (MoE) training on NVIDIA GB200 hardware, raising DeepSeek-V3 performance from 103 to 1,068 TFLOPS/GPU.

By Illumora Editorial

Source · Sep 14, 2026, 4:39 PM · On Illumora · Sep 14, 2026, 4:42 PM

Media from the primary source — shown here so you can stay on Illumora.

Rewritten from one allowlisted primary — not independent enterprise reporting. Lanes →

Brief drafted by Illumora’s editorial model from the linked primary source. Ops desk reviews flagged pieces. How we write →

Read the source →NVIDIA Developer Blog — Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine | NVIDIA Technical Blog
Save

NVIDIA has detailed how its Transformer Engine library, when integrated with JAX, significantly enhances the performance of Mixture of Experts (MoE) model operations. This optimization specifically targets dropless MoE training, which processes every token without dropping or padding to preserve model quality.

The improvements were demonstrated during DeepSeek-V3 training on NVIDIA GB200, where throughput increased by 10.4x.

Key Points

  • NVIDIA Transformer Engine with JAX improved MoE training throughput by 10.4x on NVIDIA GB200.
  • DeepSeek-V3 performance rose from 103 to 1,068 TFLOPS/GPU with these optimizations.
  • Dropless MoE processes every token, avoiding drops or padding, which helps maintain model quality.
  • Transformer Engine uses grouped GEMM kernels to handle variable expert token counts in a single call.
  • Expert parallelism operations are accelerated by NCCL EP, which fuses dispatch and combine stages and deduplicates tokens.
  • The full stack achieved 97% scaling efficiency at 1,024 GPUs on NVIDIA GB300 NVL72 hardware when training DeepSeek-V3 671B.

Context

According to the NVIDIA Developer Blog, Mixture of Experts (MoE) has become a defining architectural trend in large-scale AI model training, with examples including DeepSeek, Qwen, and Mixtral. MoE models offer efficient training through conditional computation, replacing a single dense feed-forward network with multiple smaller expert networks and a learned router. However, scaling MoE training efficiently presents challenges, particularly with token routing, expert dispatch and gather, all-to-all communication, and ragged expert GEMMs.

Why It Matters

This advancement addresses critical bottlenecks in large-scale MoE training, enabling developers to train more complex and higher-quality models without compromising efficiency. The ability to handle dropless MoE efficiently means that models can leverage the full input data, potentially leading to more robust and accurate AI systems.

What To Do

  • Try the NVIDIA NGC MaxText container with Transformer Engine enabled to reproduce the optimized JAX MoE path.
  • Read the MaxText MoE Configuration guide for detailed setup instructions.
  • Review the Transformer Engine documentation to understand the library's capabilities.

Keep Exploring

/atlas/claude-family /atlas/gpt-family /atlas/**gemini**-family /atlas/llama-open