Boa preprint reports faster context compression for language models

Researchers report that Boa, a context compressor built on a Mamba state-space model, matches or surpasses leading offline compression methods while running an order of magnitude faster than Transformer-based implementations. The preprint says the approach reduces end-to-end inference latency rather than adding compression overhead. Boa processes context in a single pass and produces a shorter representation for a frozen language model. Training teaches it to reproduce the output distribution obtained from the full context. The researchers demonstrate compression ratios as aggressive as 16x; the reported performance comparison applies at equal compression ratios.