Region-aware KV cache eviction cuts memory pressure for LLM agents with heterogeneous context

Researchers proposed MemDecay, a training-free, region-aware KV-cache eviction policy for LLM agents that accumulate heterogeneous context including system instructions, plans, user turns, retrieved documents, tool outputs, and intermediate reasoning. Existing eviction policies apply the same attention or recency rule to every token, ignoring semantic structure. MemDecay assigns tokens region-specific base priorities and decay rates, refreshes retention scores when tokens receive attention, and evicts the lowest-scoring pages under a fixed cache budget while allowing critical regions to be pinned. A calibration procedure derives decay rates from measured attention lifetimes. Evaluated at roughly 450 and 1,700 token contexts using Qwen2.5-1.5B and 3B, attention lifetimes differed by an order of magnitude across regions: system-token half-lives ranged from 148 to 189 decoding steps versus 14 to 16 for scratchpad tokens. Pinning preserved system-region facts at full-cache accuracy in every setting; no baseline preserved more than 13 of 24. Region-aware retention remained effective as context grew, whereas recency-based retention collapsed. Accumulated-attention retention performed better on unpinned content; attention-score normalization was identified as the main limitation.