Preprint reports more stable LoRA training with revised initialization

Researchers report that changing how LoRA adapters are initialized makes language model fine-tuning more stable when their two matrices use different learning rates. Their preprint finds that the change prevents performance collapse at high learning-rate ratios across tasks and models. LoRA adapts a model through smaller trainable matrices. The researchers trace instability to noise introduced by the standard initialization and propose reducing the starting variance of one matrix while keeping the other at zero. With adapter rank and training time matched, the revised method widens the stable learning-rate region. The findings concern training stability rather than a quantified reduction in fine-tuning costs.