MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

Jul 16, 2026

Upload by Miles

MemDecay delivers a training-free KV-cache eviction framework for long-horizon LLM agents whose traces accumulate system instructions, plans, user turns, tool outputs, retrieved documents, and intermediate reasoning. Existing eviction methods score tokens via attention or recency and apply uniform retention, ignoring the semantic region structure of orchestrated prompts. MemDecay labels tokens by prompt region, combines region-specific base priority with attention-derived importance, and applies region-specific temporal decay under a fixed cache budget, evicting the lowest-priority pages while protecting pinned instruction regions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

Abstract

Introduction

Motivation: KV cache limits for agentic traces

Proposed approach: region-aware decay and eviction

Measurement and findings (overview)

MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference Venkatesha Matam Independent Researcher New York City, USA[matam.venkatesha@gmail.com](matam.venkatesha@gmail.com) Keon Kim Independent Researcher New York City, USA [kwk236@gmail.com](kwk236@gmail.com) arXiv :2607 . 10582v 1 [ cs .LG] 12 Jul 2026 Abstract—Large language models (LLMs) are increasingly deployed as agents whose contexts accumulate system instructions, plans, user turns, tool outputs, retrieved documents, and intermediate reasoning over many steps. The key–value (KV) cache that supports autoregressive attention grows with this trace, and its memory footprint constrains long-horizon inference and serving. Existing KV-cache eviction policies score tokens with attention or recency signals and apply the same retention rule to the entire sequence, leaving the semantic structure of orchestrated agent prompts unused. We propose MemDecay, a training-free KV-cache eviction framework for long-horizon LLM agents. MemDecay assigns each token a region label supplied by the orchestration layer, combines region-specific base priorities with attention-derived importance, and applies region-specific temporal decay to each token’s retention priority. Under a fixed cache budget, MemDecay evicts the pages holding the lowest-priority tokens while protecting pinned instruction regions. Region labels align with contiguous prompt spans, so the policy maps onto page-granular cache layouts in existing serving stacks. We further describe a measurement protocol that estimates region-conditioned attention lifetimes from agent traces and uses them to calibrate decay rates. In a controlled study spanning two context lengths (roughly 450 and 1,700 tokens) and two model sizes (Qwen2.5-1.5B and 3B), measured lifetimes differ by an order of magnitude across regions in every setting (system halflives of 148 to 189 decode steps against 14 to 16 for scratchpad reasoning), and the ordering of the movable regions is robust to insertion order. The eviction study yields one clear win and one instructive loss. Pinning preserves system-region facts at the fullcache ceiling in every setting, no baseline preserves more than 13 of 24, and recency-based retention collapses entirely as contexts grow while the structural prior keeps working. On unpinned content, accumulated-attention retention strengthens with model scale and wins; an ablation shows the policy’s attention term cannot compensate at any tested weight, pointing to magnitude normalization of that term as the needed design change. Index Terms—KV cache, KV cache compression, cache eviction, large language models, LLM agents, agentic workflows, efficient LLM inference, LLM serving, memory management, long-context inference I. INTRODUCTION Large language models (LLMs) are increasingly used in longhorizon agentic workflows that combine multi-turn interaction, retrieval, tool use, and iterative planning. Unlike single-pass long-context tasks, these systems accumulate a persistent execution trace composed of system instructions, safety constraints, evolving plans, user turns, retrieved evidence, tool interactions, and intermediate reasoning states. The trace is structured: its components serve different functional roles and should persist for different durations. This shift places new pressure on the key–value (KV) cache that supports autoregressive decoding. Because the KV cache grows with sequence length, long interaction traces translate into substantial memory overhead, increased bandwidth demand, and higher end-to-end latency [1], [2] . Prior work reduces this overhead through selective retention and compression, including policies that adapt eviction to stable differences in attention behavior across heads [3], [4] . Most of these approaches target generic long-context generation and score cached tokens with model-centric signals such as accumulated attention, attention sinks and recency windows, or i

FAQs

What problem does MemDecay address in LLM agent inference?

Long-horizon agent traces grow the KV cache, increasing memory, bandwidth, and latency. Standard eviction rules treat all tokens uniformly and do not preserve the semantic roles needed over different time scales.

How does MemDecay decide which KV-cache pages to evict?

Each token receives a region label from the orchestration layer, then MemDecay computes a composite retention score using region-specific base priority, attention-derived importance, and region-specific temporal decay. It evicts pages containing the lowest-priority tokens under a fixed cache budget while protecting pinned instruction regions.

What does the study show about attention lifetimes across regions?

Region-conditioned attention lifetimes differ by about an order of magnitude across regions in every tested setting. The ordering of movable regions remains robust, and pinning preserves system-region facts while recency-only retention deteriorates as contexts grow.

Jul 16, 2026

Upload by Miles