# BitSparse preprint reports lower training memory use with lossless compression

_Published Friday, October 16, 2026 at 5:22 AM EDT · AI, Science · Latest · Tier 2 — Notable_

Researchers report that BitSparse reduced per-token GPU memory use by 31% in Modded-NanoGPT and 30% in Nemotron-H, with overheads of no more than 1% at practical sequence lengths. The preprint introduces a lossless format for storing sparse activations, the intermediate values retained during model training.

Custom GPU kernels compress those values and reconstruct them exactly for the backward pass used to calculate training updates. The format supports BF16 and FP8 activations, with optional sign-bit packing for additional compression. Its approach targets sparse ReLU and squared-ReLU activations.

## Sources

- [cs.LG updates on arXiv.org](https://arxiv.org/abs/2610.22256)

---
Canonical: https://techandbusiness.org/newswire/8u8w22NUsOp7w7i0HkkBaN
Published: 2026-10-16T09:22:51.104Z
Story chronology: 2026-10-16T04:00:00.000Z
Retrieved: 2026-10-16T13:02:53.680Z
Publisher: Tech & Business (techandbusiness.org)