Smaller, faster, safer: running Kimi and GLM at scale

Cloudflare said it has deployed three optimization techniques to run Moonshot's Kimi K-series and Z.ai's GLM mixture-of-experts models on its Workers AI platform. The company quantizes the key-value cache to 8-bit floating point, compresses model weights to 4-bit integers, and adds integrity checks to protect the shared cache. Quantizing the KV cache on Kimi K2.6 doubles the context capacity to about 1.37 million tokens and allows 64 concurrent requests instead of 32. Cloudflare said this reaches 2,192 tokens per second, about 41% higher than the 16-bit precision peak, for roughly 30% less cost per token. Compressing GLM 5.2 weights from 8-bit to 4-bit integers shrinks the checkpoint from 705 GB to 421 GB and reduces per-GPU memory from 88 GB to 52 GB. The company said decode speed improves because less data moves across memory bandwidth, while prefill remains in higher precision to avoid a slowdown. Cloudflare added a KV cache integrity checking layer that tags physical cache pages and validates mappings before decode operations read from the cache. The company said the safety check costs under 1% on both throughput and tail latency in production measurements. All work runs on the SGLang inference serving framework.