ReCoLoRA Method Recycles LoRA Adapters to Reduce Catastrophic Forgetting in Continual Fine-Tuning

A new method called ReCoLoRA addresses the problem of catastrophic forgetting when sequentially fine-tuning large language models with low-rank adapters. Standard LoRA stacks new adapters on frozen weights, causing each task to overwrite the previous one. ReCoLoRA instead re-decomposes the current effective weight before each new task into a frozen residual, a slowly updated principal component, and a fresh adapter initialized from a randomized SVD of the pretrained weight with per-layer ranks chosen by an elbow criterion. On a six-task continual GLUE sequence across four 7-8B backbones, ReCoLoRA achieved the best final average score on three of four models against rank-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines while training fewer parameters. An oracle-routed task-bank variant provides an upper bound under full task isolation.