# AWS reports higher MoE inference throughput from available software optimizations

_Published Tuesday, October 13, 2026 at 11:32 AM EDT · AI · Latest · Tier 2 — Notable_

![AWS reports higher MoE inference throughput from available software optimizations — Primary](https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2026/10/08/ML-22058-featured-image.png)

AWS says software optimizations available for serving mixture-of-experts models delivered up to 10 percent higher throughput and up to 9 percent faster time to the first generated token. These models route tokens through selected specialized networks to expand capacity while limiting computation.

Routing FP8 models to the Triton backend produced a 10 percent throughput improvement on Nvidia H200 GPUs; that configuration is available in AWS's Large Model Inference v28 container. Separate changes to matrix layout and unnecessary kernel recompilation are available in vLLM versions 0.24.0 and 0.22.0 or later, respectively. The reported gains depend on specific workloads and hardware; the first-token result used H200 GPUs with computation split across eight GPUs.

## Sources

- [Artificial Intelligence](https://aws.amazon.com/blogs/machine-learning/mixture-of-experts-kernel-engineering-on-amazon-sagemaker-ai/)

---
Canonical: https://techandbusiness.org/newswire/i2777jMYmix5yjeZnwjx12
Published: 2026-10-13T15:32:04.728Z
Story chronology: 2026-10-13T15:17:09.000Z
Retrieved: 2026-10-13T17:51:12.042Z
Publisher: Tech & Business (techandbusiness.org)