Multi-Head Recurrent Memory Agents - arXiv 2607.01523 - Abstract and Introduction

Sep 21, 2026

Recurrent memory agents extend LLMs to arbitrarily long contexts by iteratively consolidating input into a fixed-size memory window, but end-to-end reliability degrades as context grows. The work separates performance into memory capture and memory retention and finds retention collapses as the dominant bottleneck. Existing designs store memory as a monolithic text block, so updates risk overwriting retained information. To address this, a training-free Multi-Head Recurrent Memory (MHM) framework partitions memory into independent heads with select-then-update shielding, instantiated as MHM-LRU.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Abstract

1 Introduction

arXiv :2607 .0 1523v 1 [ cs .LG] 1 Jul 2026 Multi-Head Recurrent Memory Agents Jiatong Li Samuel Yeh Sharon Li Department of Computer Science, University of Wisconsin-Madison {jli2947, samuelyeh, [sharonli}@cs.wisc.edu](sharonli}@cs.wisc.edu) Abstract Recurrent memory agents extend LLMs to arbitrarily long contexts by iteratively consolidating input into a fixed-size memory window. Despite their scalability, these agents exhibit a well-documented reliability problem: end-to-end performance degrades systematically as context length grows. We diagnose this failure by decomposing performance into two factors—memory capture and memory retention—and quantitatively confirm that retention is the dominant bottleneck. Retention collapses because existing designs maintain memory as a monolithic text block, forcing every update to risk overwriting previously retained content. Motivated by this diagnosis, we propose Multi-Head Recurrent Memory (MHM), a general, training-free framework that partitions memory into independent heads governed by a stage-wise select-then-update strategy. At each step, exactly one head is selected for update while the remaining heads are structurally shielded from overwriting, shifting the burden of retention from model behavior to architectural design. As a lightweight instantiation, we introduce Least-Recently-Updated MHM (MHM-LRU), which guarantees uniform head utilization with zero additional token overhead. Extensive experiments on long-context benchmarks show that MHMLRU substantially improves both retention and end-to-end accuracy across the 100K–1M token range, where baselines degrade sharply. On RULER-HQA at 896K tokens, MHM-LRU improves the memory retention rate from less than 30% to 73.96% . These gains generalize across model families, scales, and task types, positioning architectural optimization as a practical and cost-efficient path toward reliable long-context recurrent memory. Our code is available here. 1 Introduction The ability to reason over long contexts is a fundamental requirement for LLM agents operating in the real world [1–5] . Tasks such as document analysis [6–9], multi-step research [10–12], and extended dialogue [1, 13] demand that agents track and integrate information across inputs that far exceed the context window. Recurrent memory [14–16] addresses this challenge by processing input sequentially in chunks and iteratively compressing a long context into a fixed-size memory window. At each step, the agent reads a new context chunk and updates its memory to integrate new information with what has been retained so far. This paradigm is appealing because it places no hard limit on context length and requires no modification to the underlying LLM. In practice, however, recurrent memory agents exhibit a well-documented reliability problem: performance degrades systematically as context length grows [17, 18] . To motivate our work, we first diagnose how existing recurrent memory fails under long contexts by decomposing end-to-end performance into two factors: whether the memory ever captures the relevant information during memory update, and whether it retains that information through all subsequent memory updates. Measuring both memory capture rate and memory retention rate on a representative method MemAgent [17], we find that capture remains stable across all context lengths, while memory retention decreases sharply, falling below 30% at 896K tokens. This diagnosis Preprint. Figure 1: Left: An overview of the multi-head recurrent memory (MHM) . Right: An overview of the memory head utilization pattern across time. In every time step, only one head is written. As time goes by, MHM achieves asymptotic uniform memory utilization. pinpoints memory retention failure as the dominant bottleneck. The root cause is architectural: existing recurrent memory designs maintain memory as a monolithic text block, so every update risks erasing key information. Information w

FAQs

What problem do recurrent memory agents face with long contexts?

Their end-to-end performance degrades systematically as context length grows.

How does the paper explain the reliability failure?

It decomposes performance into memory capture and memory retention and shows retention is the dominant bottleneck.

What is Multi-Head Recurrent Memory (MHM) and how does it help?

MHM partitions memory into independent heads and uses a stage-wise select-then-update strategy so unselected heads are structurally protected from overwriting, improving retention and downstream accuracy.

Sep 21, 2026

Uploaded by Ava Thompson