LaProx researchers report preserving model performance with 5% of inference cache

Researchers introducing LaProx report that their method preserved language model performance while retaining only 5% of the cache used to process long inputs. The preprint reports experiments across 19 datasets on the LongBench and Needle-In-A-Haystack benchmarks. The cache stores intermediate information used during inference. LaProx selects which tokens to retain by estimating their contributions to model outputs and comparing importance across the model, rather than making separate decisions within individual attention heads. The researchers report up to 2× reduction in accuracy loss against existing leading methods under extreme compression; the findings concern benchmark experiments.