TriAttention preprint reports faster generation with selective attention precision

Researchers behind the TriAttention preprint report end-to-end language model generation speedups of up to 7.88x over Torch attention and 2.65x over a BF16 baseline, while maintaining comparable quality on general-purpose and long-context tasks. The method assigns blocks of attention calculations to skipping, 4-bit or 8-bit computation according to calibration measurements and a shared computation budget. It also reuses compressed keys and values during decoding, reducing repeated reads of prompt data. The reported gains come from the authors' experiments.