AWS launches GPU-aware SageMaker HyperPod Inference Gateway

AWS announced general availability of the SageMaker HyperPod Inference Gateway, a Kubernetes-native routing addon that places inference requests using real-time GPU signals such as KV cache utilization, queue depth, LoRA adapter residency and prefix cache hit rate. AWS says the addon installs on existing HyperPod/EKS clusters with no application or model-server changes and exposes an OpenAI-compatible endpoint. In AWS benchmarks across models from 8B to 235B parameters, the company reported first-token latency reductions of up to 82 percent versus a Kubernetes round-robin baseline, with gains concentrated in mixed-GPU, bursty and shared-prefix workloads. A second-tier Global Inference Router for cross-cluster routing is described as coming soon.