AWS makes GPU-aware inference routing available on SageMaker HyperPod

AWS says its SageMaker HyperPod Inference Gateway is available for routing model requests within a cluster in supported regions. It installs as an EKS managed add-on on existing HyperPod infrastructure and works with OpenAI-compatible model servers without application code changes. The gateway reads the requested model and selects a GPU server using signals including queue depth and cache use. AWS says this approach cuts first-token latency by up to 82% and reduces p99 first-token latency by 97-98% in mixed-hardware and burst-traffic scenarios. Routing across clusters and regions is still planned.