AWS adds prefix-aware routing to SageMaker Inference for LLM endpoints

Amazon SageMaker Inference introduced prefix-aware routing, a routing strategy that sends requests sharing the same prompt prefix to the same instance so cached key-value pairs are reused instead of recomputed. AWS said the feature is available today on real-time inference endpoints and is configured per production variant via RoutingStrategy and PrefixAwareRoutingConfig, switchable without redeploying the model. In AWS benchmarks on Llama 3.1 70B across seven ml.p5.48xlarge instances with vLLM prefix caching enabled, P50 time-to-first-token fell by up to 77 percent and throughput rose up to 16 percent on long-context workloads, with KV cache hit rates moving from roughly 25 percent to over 80 percent. AWS reported routing overhead of 1.3-1.9 milliseconds per request.