Preprint reports lower distillation costs using a fraction of teacher scores

A preprint reports that retaining 0.01% of a teacher language model's vocabulary scores can reduce the cost of training a smaller model without observed MMLU-Pro degradation in tested Qwen3 settings. Distillation trains a student model to reproduce a teacher's outputs. Caching the retained next-token scores cut training step time by 38%, while restricting the student's output computation to those tokens cut peak memory by 24%. Neither change caused observed MMLU-Pro degradation. The cutoff depended on the model family. Nemotron Nano matched full-vocabulary MMLU-Pro performance at 0.1%, but smaller fractions failed to emit the closing reasoning tag. Agreement with the teacher's complete probability distribution also worsened as support shrank.