TaSQ preprint reports higher AI throughput with 1-bit cache compression

Researchers report in a preprint that TaSQ, a method for compressing the memory used during language-model inference, supports up to 14× larger batch sizes and achieves 1.87× higher peak throughput than a BF16 baseline on a single RTX 6000 Ada GPU. The results come from an implementation in SGLang. TaSQ compresses the key-value cache, which stores information used to process subsequent tokens. It weights and groups cached channels to reduce compression errors, with transforms that can be merged into model weights and compression codebooks. The researchers report better results than existing low-bit baselines across general, reasoning and long-context retrieval benchmarks; the stated throughput comparison is specific to that GPU implementation.