From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

Jul 16, 2026

KV cache has become a central memory object in autoregressive LLM serving, especially under long-context and high-concurrency workloads where per-request KV state can reach tens of gigabytes and aggregate to terabyte scale. This survey classifies 30+ KV-management systems and frameworks using locality, lifetime, ownership, and substrate, yielding five architectural archetypes. It audits existing evaluations and highlights seven missing KV-specific measurements tied to fault tolerance, isolation, eviction, speculative decoding, MoE serving, and shared-cache semantics.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages34

Reading Time1 h 26 min

Introduction

Why the Local KV Abstraction Is No Longer Sufficient

Six Design Responses

Thesis

arXiv :2607 .02574v1 [ cs .DC] 30 Jun 2026 From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving JIE LI, Texas Tech University, USA TONGYANG WANG, Texas Tech University, USA YONG CHEN, Texas Tech University, USA The key-value (KV) cache has become a first-order memory object in LLM serving rather than a temporary per-request tensor. This survey classifies more than thirty KV-management systems and frameworks using four axes: locality, lifetime, ownership, and substrate. The axes reveal five architectural archetypes—local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier. Once workload and hardware are fixed, ownership accounts for much of the remaining design variance among distributed systems. The survey also audits current evaluations and identifies seven missing KV-specific measurements, linking them to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics. CCS Concepts: • Computer systems organization → Distributed architectures; Cloud computing; • Computing methodologies → Artificial intelligence. Additional Key Words and Phrases: KV cache, LLM serving, distributed systems, inference optimization, memory hierarchy 1 Introduction The key-value (KV) cache is a major dynamic memory consumer in autoregressive LLM inference [9, 48, 64], particularly for long-context and high-concurrency serving. For a decoder-only model with 􀀡 layers, 􀀝kv KV attention heads, head dimension 􀀙, sequence length 􀀨, and 􀀱 bytes per KV element, the aggregate per-request KV cache footprint is 2 × 􀀡 × 􀀝kv × 􀀙 × 􀀨 × 􀀱 bytes, before accounting for tensor-parallel sharding, allocator fragmentation, metadata, or paging overheads. The factor of two accounts for separately stored keys and values. For a Llama-3.1-70B-style model with 􀀡 = 80,􀀝kv = 8, 􀀙 = 128, and a 128K-token context, this corresponds to roughly 40 GiB per request in BF16/FP16 . At 32K context, each request still requires about 10 GiB, so one hundred concurrent requests approach 1 TiB of aggregate KV state. As workloads shift toward long-context prompting, multi-turn interaction, and retrieval-augmented generation (RAG), the KV cache has become a first-order serving bottleneck rather than a small temporary buffer confined to the producing GPU. This survey argues that the community is in the middle of an architectural shift. The KV cache was once a local tensor freed at request completion. Modern systems increasingly treat it as an explicit scheduling and memory-management object. Some systems remain local but virtualize, schedule, evict, or compress KV to raise effective capacity. Others move KV across workers, requests, memory tiers, or pooled-memory substrates, making it a distributed memory tier with explicit placement, lifetime, ownership, and substrate policies. These mechanisms are related responses to four questions: where the cache lives, how long it survives, who owns it, and what carries or exposesit. The survey’s organizing contribution is to show that the classified systems, frameworks, and KV-reduction techniques—roughly three dozen entries enumerated in Tables 8 and 9—concentrate around five recurring archetypes when sorted by their answers. This research is supported in part by the National Science Foundation under grant OAC-2404438, and CNS-1939140 (A U.S. National Science Foundation Industry-University Cooperative Research Center on Cloud and Autonomic Computing) . Authors’ Contact Information: Jie Li, [jie.li@ttu.edu](jie.li@ttu.edu), Texas Tech University, Department of Computer Science, Lubbock, Texas, USA; Tongyang Wang, [tongyang.wang@ttu.edu](tongyang.wang@ttu.edu), Texas Tech University, Department of Computer Science, Lubbock, Texas, USA; Yong Chen, [yong.chen@ttu.edu](yong.chen@ttu.edu), Texas Tech University, Department of Computer Science, Lubbock, Texas, USA. 2 Jie Li, Tongyang Wang, and Yong Chen 1.1 W

FAQs

Why is the KV cache no longer sufficient as a per-request local abstraction in modern LLM serving?

Longer context windows, multi-turn/retrieval reuse, and agentic workflows push KV state across boundaries such as GPUs, nodes, memory tiers, and request sessions. Serving systems must define the semantics and cost of each crossing rather than treating KV as purely per-request transient memory.

How does the survey classify KV-management systems?

Systems are organized along four axes: locality, lifetime, ownership, and substrate. These axes lead to five recurring architectural archetypes: local-paged, disaggregated-pipeline, shared-store, memory-pool, and hybrid-tier.

Which evaluation gaps does the survey identify for KV-cache research?

The survey audits current evaluations and identifies seven missing KV-specific measurements. These gaps are linked to open problems in fault tolerance, isolation, tiered eviction, speculative decoding, MoE serving, and shared-cache semantics.

Jul 16, 2026

Upload by Emma Wilson