Researchers report 2-bit LLaMA2-70B compression with 1% accuracy drop

Researchers propose a method for compressing language models that achieved 2-bit quantization of LLaMA2-70B with a 1% accuracy drop on popular question-answering benchmarks, according to their arXiv preprint. The method, called TVQ, trains compressed model weights to recover quality lost during compression. Vector quantization reduces the memory needed to store model weights, but its discrete parameters make further optimization difficult. TVQ introduces trainable latent weights that allow gradient-based optimization and uses block-wise knowledge distillation to balance quality and training efficiency. The reported accuracy result applies to the named model and benchmark category.