# arXiv paper proposes topology-aware KV cache moves for disaggregated GPU inference

_Monday, August 3, 2026 at 12:00 AM EDT · Science, AI · Latest · Tier 2 — Notable_

An arXiv preprint titled Topology-Aware Data Movement for Disaggregated GPU Inference argues that splitting LLM prefill and decode across separate GPU pools creates a networking problem existing systems do not handle correctly. For a 70B model the authors put KV cache transfer at 2.6 GB per request, exceeding 100 GB/s aggregate at production scale.

They write that DistServe, Splitwise, and Mooncake all use uniform RDMA even though bandwidth between two GPUs can vary by 72x with physical relationship: 900 GB/s via NVLink within a domain, 50 GB/s via InfiniBand across nodes, and 12.5 GB/s via TCP across data centers. The paper proposes a topology-aware transfer orchestrator that maps interconnect hierarchy at startup and picks transport per transfer.

Three mechanisms are described together: pipelined layer-by-layer transfer that overlaps transmission with ongoing prefill and is projected to hide 60 to 85 percent of latency behind computation; NVLink domain-aware placement for Mixture-of-Experts models that co-optimizes expert dispatch with KV cache locality; and CXL 3.0 memory expanders as a shared overflow tier providing 6x capacity at 86x lower latency than NVMe. Full multi-node evaluation with heterogeneous interconnects and CXL 3.0 is described as beyond current academic and cloud availability; analytical models and component work project 3 to 18x transfer latency reduction versus uniform RDMA.

## Sources

- [arXiv](https://arxiv.org/abs/2607.28633)

---
Canonical: https://techandbusiness.org/newswire/Ymz0-G4ZR0JsDVArASx3TS
Retrieved: 2026-08-03T14:47:24.085Z
Publisher: Tech & Business (techandbusiness.org)