ReCoLoRA Method Recycles LoRA Adapters to Reduce Catastrophic Forgetting in Continual Fine-Tuning

A new method called ReCoLoRA addresses the problem of catastrophic forgetting when sequentially fine-tuning large language models with low-rank adapters. Standard LoRA stacks new adapters on frozen weights, causing each task to overwrite the previous one. ReCoLoRA instead re-decomposes the current effective weight before each new task into a frozen residual, a slowly updated principal component, and a fresh adapter initialized from a randomized SVD of the pretrained weight with per-layer ranks chosen