Preprint reports faster AI serving after redesigning remote cache loading

Researchers report that redesigning a remote store for reusable model calculations cut p99 latency from 83.32 seconds to 0.67 seconds on a request stream dominated by cache hits. The preprint studied GLM-5.3 on 8xB300 nodes, where loading cached calculations could take longer than recomputing them. The changes split storage across processing ranks, loaded data asynchronously and fixed three client bugs. The sharded store increased prefill throughput by 14%, or 34% with an upstream kernel fix. All changes have been submitted to LMCache. The results concern this deployment and request stream; a fixed loading cost still limited the benefit for shorter prompts.