arXiv paper proposes topology-aware KV cache moves for disaggregated GPU inference

An arXiv preprint titled Topology-Aware Data Movement for Disaggregated GPU Inference argues that splitting LLM prefill and decode across separate GPU pools creates a networking problem existing systems do not handle correctly. For a 70B model the authors put KV cache transfer at 2.6 GB per request, exceeding 100 GB/s aggregate at production scale. They write that DistServe, Splitwise, and Mooncake all use uniform RDMA even though bandwidth between two GPUs can vary by 72x with physical relationship: 900 GB/s via NVLink within a domain, 50 GB/s via InfiniBand across nodes, and 12.5 GB/s via TCP across data centers. The paper proposes a topology-aware transfer orchestrator that maps interconnect hierarchy at startup and picks transport per transfer. Three mechanisms are described together: pipelined layer-by-layer transfer that overlaps transmission with ongoing prefill and is projected to hide 60 to 85 percent of latency behind computation; NVLink domain-aware placement for Mixture-of-Experts models that co-optimizes expert dispatch with KV cache locality; and CXL 3.0 memory expanders as a shared overflow tier providing 6x capacity at 86x lower latency than NVMe. Full multi-node evaluation with heterogeneous interconnects and CXL 3.0 is described as beyond current academic and cloud availability; analytical models and component work project 3 to 18x transfer latency reduction versus uniform RDMA.