# Preprint reports lower distillation costs using a fraction of teacher scores

_Published Saturday, November 14, 2026 at 9:06 AM EST · AI, Science · Latest · Tier 2 — Notable_

A preprint reports that retaining 0.01% of a teacher language model's vocabulary scores can reduce the cost of training a smaller model without observed MMLU-Pro degradation in tested Qwen3 settings. Distillation trains a student model to reproduce a teacher's outputs.

Caching the retained next-token scores cut training step time by 38%, while restricting the student's output computation to those tokens cut peak memory by 24%. Neither change caused observed MMLU-Pro degradation.

The cutoff depended on the model family. Nemotron Nano matched full-vocabulary MMLU-Pro performance at 0.1%, but smaller fractions failed to emit the closing reasoning tag. Agreement with the teacher's complete probability distribution also worsened as support shrank.

## Sources

- [cs.AI updates on arXiv.org](https://arxiv.org/abs/2611.14850)

---
Canonical: https://techandbusiness.org/newswire/9gtjOiwhK_Oo6sB3gfbh0W
Published: 2026-11-14T14:06:56.141Z
Story chronology: 2026-11-14T05:00:00.000Z
Retrieved: 2026-11-15T00:01:50.053Z
Publisher: Tech & Business (techandbusiness.org)