Muon-RT preprint reports lower training memory and higher accuracy in self-distillation tests

Researchers propose Muon-RT, a language-model training method that reuses information already retained by a teacher model to reduce persistent memory. Their arXiv preprint reports that, in self-distillation fine-tuning of Qwen2.5-7B-Instruct, it improved average accuracy over AdamW by 2.6 points with 42% less persistent memory, and over momentum Muon by 5.4 points with 27% less persistent memory. The method eliminates separate momentum buffers for hidden weight matrices by using the displacement between teacher and student models. Its theoretical convergence result depends on explicit regularity and alignment assumptions.