Google expands Kubernetes tools for AI training and distributed inference

Google Cloud introduced and expanded an open-source stack for running AI workloads on Kubernetes, including OpenRL for reinforcement learning and llm-d 1.0 for distributed inference. Google says it created OpenRL and contributed it to the CNCF Sandbox. OpenRL interleaves training and model-generated practice runs on the same accelerators and sends only changed model parameters to workers. The approach lets teams share hardware instead of maintaining separate pools for those tasks. The llm-d software routes requests toward workers holding reusable model context. Google reports that production deployments increased cache hit rates from 35% to 70% and cut median time to first token by 70-85% on multi-turn agent workloads. Those performance figures apply to the cited deployments.