Recent breakthroughs in context-to-weight compilation, notably Doc-to-LoRA and SHINE,demonstrate that the Transformer Key-Value (KV) cache bottleneck can be bypassed by utilizingHypernetworks to compile textual context directly into transient Low-Rank Adaptation (LoRA)matrices. However, these early architectures suffer from two critical flaws: (1) computationalintractability when scaling to multi-million token contexts, as Hypernetworks cannot processinfinite sequences efficiently, and (2) the Stability-Plasticity Dilemma, where aggressive factualweight injection catastrophically corrupts the base model’s linguistic manifolds. In this paper,we introduce the Macular Ephemeral State-space Architecture (MESA), which resolves bothlimitations. MESA utilizes a linear-time System-1 dense retriever (The Macula) to reduce Hypernetwork input processing from O(N2) to O(1). Furthermore, we introduce a Dual-ObjectiveKL-Divergence regularization during the meta-learning phase. Our empirical results prove thatMESA maintains a flat ∼ 1.05 GB VRAM footprint regardless of context length, and our regularization successfully drives KL divergence down to 0.0588, achieving high factual plasticitywhile preserving mathematically perfect base-model generative grammar.
Eun Jung Lee (Thu,) studied this question.