HAServe preprint reports higher throughput from coordinated scheduling and memory

Researchers report that HAServe, a method for coordinating request scheduling and memory retention in language model services, increased throughput by up to 1.08-1.44× across evaluated workloads. The preprint also reports cache hit rates up to 20.59 percentage points higher than the strongest baseline for each metric and setting. HAServe jointly chooses which requests to admit and which cached intermediate results to retain, using visible demand to minimize predicted latency. Researchers implemented it on SGLang and tested Qwen3-4B-Instruct-2507 and Llama2-13B across five workloads using eight H100 GPUs. The reported gains apply to those evaluated models, workloads and hardware.