SlimKV preprint reports faster decoding with compressed AI memory

Researchers report in a preprint that SlimKV, a method for compressing the memory language models retain from earlier tokens, achieved up to 3.38x end-to-end decoding speedup over an uncompressed model at 128K context length. The method trains models to store compressed context representations and perform attention directly on them, reducing the latency of reconstructing full-sized representations. On LongBench, it retained over 96% of the uncompressed model's score at 4x/8x compression and outperformed baselines at 16x/32x compression. The reported speedups are evaluation results, with attention alone accelerating by up to 7.34x.