Google tests global GKE inference routing across 17,000 nodes

Google says a multi-cluster GKE Inference Gateway test distributed foundation-model traffic across 17,000 compute nodes in three U.S. and European regions while retaining 99.5% of direct local-cluster throughput. Expanding from one cluster to three produced a near-linear throughput increase and a 99.9% request success rate under heavy concurrency. The gateway used live key-value cache utilization, a measure of model-memory pressure, to spill requests from saturated regions to healthy ones behind one global address. The company tested one mixture-of-experts model with SGLang, so the results do not establish equivalent performance for other models or workloads.