KnitLoRA: bridging low-rank adaptation as interwoven layers for deeper semantic reasoning

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method that facilitates the lightweight adaptation of large language models (LLMs) by introducing low-rank update matrices, and has since motivated the development of various extensions. However, LoRA and its most variants overlook the issue of information loss in deep LoRA layers, where signals or gradients passing through multiple LoRA blocks may gradually vanish before reaching the final layers of the network. This limitation hampers convergence speed during fine-tuning. To address these challenges, we propose KnitLoRA, an innovative dense connection low-rank adaptation framework. First, we introduce dense connections between each LoRA block and multiple, or even all preceding blocks to facilitate feature reuse and fusion. Second, these connections improve gradient flow and mitigate the vanishing gradient problem. Third, KnitLoRA accelerates model convergence, further enhancing its efficiency in fine-tuning. Importantly, KnitLoRA eliminates dense connection paths during inference, incurring no extra computational overhead. Our experiments show that KnitLoRA outperforms LoRA and its variants without adding any additional trainable parameters, achieving stronger performance and faster, lower loss reduction. Codes and models will be available online.