Dual-System Memory Consolidation for Lifelong Learning in Language Models: Combining Direct Weight Editing with Sleep-Wake Training

We introduce a dual-system memory architecture for language models inspired by Complementary Learning Systems (CLS) theory. MEMIT (Mass-Editing Memory in Transformers) serves as fast hippocampal encoding, injecting facts directly into MLP weights during wake. LoRA fine-tuning serves as slow neocortical consolidation during sleep. We develop covariance-regularized MEMIT with cross-edit null-space constraints that prevent new edits from overwriting previous ones, and validate the dual system across 3B, 8B, and 70B parameter models. Key ablations show: (1) the dual system outperforms either component alone, (2) null-space constraints achieve perfect retention across sequential edits, and (3) the Woodbury identity enables efficient covariance regularization in N x N space rather than d x d.