Preprint reports 2.4-3.0× serving throughput with four-bit AI memory cache

Researchers report in a preprint that BitDecoding-FP4 delivers 2.4-3.0× higher peak serving throughput than a BF16 cache in tests on NVIDIA B300 hardware with Qwen3-4B and Qwen3-30B-A3B. The method compresses the key-value cache, which stores information used to generate subsequent tokens, into four-bit formats. The design combines corrections for quantization errors with changes to attention computation that reduce processing overhead. Across the models evaluated, the researchers report accuracy within 0.5 points of BF16 with NVFP4 and within 1 point with MXFP4. They say the implementation is available; the throughput figures apply to the specified hardware and two serving models.