A HIPPOCAMPUS FOR LINEAR ATTENTION - An Exact Memory for What the Recurrent State Forgets - arXiv 2607.02303

Sep 21, 2026

Uploaded by Ivy

Linear-attention and state-space language models summarize the prefix into a fixed recurrent state, achieving O(1) memory while introducing lossy exact recall: competing key–value facts overwrite earlier ones and needle retrieval degrades. HOLA (Hippocampal Linear Attention) adds a hippocampal complement by keeping the delta-rule state as compressive memory while storing a bounded exact KV cache. The cache updates via write magnitude and reads through decoupled RMSNorm-γ retrieval. At 340M parameters trained on 15B SlimPajama tokens, HOLA reduces Wikitext perplexity and improves LAMBADA, with stronger long-context needle recall up to 32k tokens.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

ABSTRACT

INTRODUCTION

arXiv :2607 .02303v 1 [ cs .AI] 2 Jul 2026 A HIPPOCAMPUS FOR LINEAR ATTENTION AN EXACT MEMORY FOR WHAT THE RECURRENT STATE FORGETS Wanyun Cui Shanghai University of Finance and Economics ABSTRACT Linear-attention and state-space language models compress the prefix into a fixedsize recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key–value associations compete, earlier facts are overwritten and needle recall degrades. Inspired by Complementary Learning Systems, we give linear attention a hippocampal complement. HOLA (Hippocampal Linear Attention) keeps the usual delta-rule state as a compressive memory and adds a bounded exact KV cache, forming a semiparametric test-time memory: the state models linearly compressible structure, while the cache stores associations that should not be forced through that state. The cache writes without a learned eviction module, keeping tokens with large β ·∥e∥, the prediction residual actually committed to the state; a decoupled RMSNorm-γ cache read then turns these exact KV pairs into sharp retrieval rather than soft averaging. At 340M parameters trained on 15B SlimPajama tokens, HOLA lowers Wikitext perplexity from 27.32 to 22.92 (−16 . 1%), below a full-attention Transformer++ (26 .88), and improves LAMBADA perplexity from 30.95 to 30.26. It also achieves the best linear in-context retrieval and remains much more robust than GDN or a matched HOLA+recency cache on RULER needle-in-a-haystack recall out to 32k tokens (16 × its training length) . ppl 30 28 26 24 22 20 (a) Wikitext perplexity ↓ | | 27.3
| 26.9
| | | | --- | --- | --- | --- | --- | | | | | | | | | | | 26.2 | | | | | | | | DN GLA GSA GDN T++ KDA HOLA recall linear baselines full attn (T++) HOLA (ours) (b) S-NIAH-1 recall vs. context context length (trained at 2k) Transformer++ GDN (no cache) HOLA+recency HOLA Figure 1: HOLA lowers perplexity and improves length-robust needle recall. (a) At 340M, HOLA reduces Wikitext perplexity from 27 .3 to 22.9 (−16 . 1%), below full-attention Transformer++(26 .9) . (b) On RULER S-NIAH-1, HOLA remains stronger than GDN and HOLA+recency as context grows to 32k tokens. 1 INTRODUCTION The human brain does not rely on a single memory. A hippocampus-centered system can record a specific, novel event in one shot and later recall it precisely (Tse et al., 2007), while a neocortexcentered system slowly distills generalizable structure across many experiences (Frankland & Bontempi, 2005); Complementary Learning Systems (CLS) theory (McClelland et al., 1995 ; Kumaran et al., 2016) argues that the two must be separate—a system built for slow, compressive generalization would suffer catastrophic interference if forced to write single facts quickly, and so cannot also serve fast, exact, one-shot memory. Efficient language models are missing exactly this complementarity. Linear attention is precisely such a “neocortex”: a superb compressor, but a leaky exact memory. DeltaNet, Gated DeltaNet (GDN), and related linear-attention / state-space models (Katharopoulos et al., 2020 ; Gu & Dao, 2023 ; Yang et al., 2024a ;c) replace softmax attention’s growing key–value cache with a fixed-size recurrent state S, summarizing all history in O(1) memory and processing a sequence in sub-quadratic time. The state is updated online—the delta rule writes the residual between each new value and the state’s current prediction—making it an efficient running summary. But a fixed-size state is a lossy memory: it holds only a bounded number of distinct key→ value associations before new writes overwrite old ones along shared key directions. This shows up exactly where it hurts: multi-item associative recall (Arora et al., 2023), passkey / needle-in-a-haystack retrieval (Mohtashami & Jaggi, 2023), and copying a specific distant token verbatim (Jelassi et al., 2024) . There, full softmax attention—which keeps every to

FAQs

Why do linear-attention and state-space language models lose exact memory?

They compress the whole prefix into a fixed-size recurrent state. When many key–value associations compete, new writes overwrite older facts along shared key directions, so earlier information is not retained precisely.

What is HOLA and how does it improve memory?

HOLA adds a hippocampal complement to linear attention. It keeps the recurrent state as compressive memory and uses a bounded exact KV cache for associations that should not be forced through the state.

How does HOLA decide which tokens to store in its exact cache?

It uses a write-magnitude eviction score: tokens retained are those whose updates most strongly committed the correction to the state, rather than those that are merely most recent.

Sep 21, 2026

Uploaded by Ivy

A HIPPOCAMPUS FOR LINEAR ATTENTION - An Exact Memory for What the Recurrent State Forgets - arXiv 2607.02303_Page 1

Open in WPS to view all 12 pages