AWS reports higher MoE inference throughput from available software optimizations

AWS says software optimizations available for serving mixture-of-experts models delivered up to 10 percent higher throughput and up to 9 percent faster time to the first generated token. These models route tokens through selected specialized networks to expand capacity while limiting computation. Routing FP8 models to the Triton backend produced a 10 percent throughput improvement on Nvidia H200 GPUs; that configuration is available in AWS's Large Model Inference v28 container. Separate changes to matrix layout and unnecessary kernel recompilation are available in vLLM versions 0.24.0 and 0.22.0 or later, respectively. The reported gains depend on specific workloads and hardware; the first-token result used H200 GPUs with computation split across eight GPUs.