Nvidia reports 10x MoE training gain with JAX Transformer Engine path

Nvidia said its optimized JAX and Transformer Engine path achieved a 10x end-to-end throughput gain training DeepSeek-V3 671B, compared with an unoptimized baseline. The company attributes the result to grouped GEMM, optimized expert-parallel dispatch and combine, MXFP8 quantization, host activation offloading and multistream collectives. Nvidia says the stack sustained 97 percent efficiency at 1,024 GPUs and ships in its NGC MaxText container. The configuration is specific to DeepSeek-V3 and requires model-specific tuning.