Parameter-Efficient Continual Learning for Scientific Text Classification: A LoRA-Based Approach With and Without Replay

ABSTRACT: Scientific literature has been spreading widely in almost every area and this has raised a dire necessity of automated classification systems that can be used with efficiency in the same area without devastating memory lapses. The paper explores parameter-efficient continual learning (CL) of text classification in science based on Low-Rank Adaptation (LoRA), a generalization of previous studies on full fine-tuning of pre-trained language models. We evaluate four models—BERT, SciBERT, BioBERT, and BlueBERT—trained sequentially on three Web of Science datasets (WoS-5736, WoS-11967, and WoS-46985) in curriculum order, under two CL configurations: LoRA-CL (without replay) and LoRA+Replay-CL (with 10% experience replay). LoRA adapters are applied to query and value projection matrices with rank r=8 and alpha=16, reducing trainable parameters to approximately 5% of the full model. A novel third input strategy combining keywords and abstracts is introduced alongside the two strategies from the baseline. We report four CL metrics—Average Accuracy (ACC), Backward Transfer (BWT), Forward Transfer (FWT), and Forgetting Measure (FM)—alongside per-task accuracy and F1 scores. Results show that LoRA-CL matches or exceeds full fine-tuning baselines on WoS-46985 while reducing trainable parameters by approximately 95%. SciBERT achieves the best overall performance across all input strategies with a peak accuracy of 87.93% on WoS-46985; BlueBERT exhibits the most severe catastrophic forgetting. The combined input strategy outperforms keywords alone on larger datasets, establishing LoRA-CL as a viable, resource-efficient alternative for scientific text classification in continual learning settings. KEYWORDS:Catastrophic forgetting, continual learning, experience replay, LoRA, parameter-efficient fine-tuning, scientific text classification, BERT, SciBERT, BioBERT, BlueBERT, Web of Science.