MosaicKV Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

Jul 16, 2026

Upload by Anda

Long-context LLM services sustain prompts with hundreds of thousands to millions of tokens, making the KV cache a dominant serving cost. Since cache size scales linearly with context length, GPU memory is quickly exhausted, batches shrink, and throughput drops. Prior KV compression often targets only sequence or only channel, limiting gains at larger windows. MosaicKV proposes dynamic two-dimensional KV compression that selects strategies per KV vector segment using non-uniform element importance, and manages compressed caches across GPU/CPU. Experiments on an H800 show up to 16× attention speedup, 3× lower memory use, and only 1.76% average accuracy loss.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Abstract

Introduction

Long-context LLM and KV-cache bottleneck

Prior KV-cache compression approaches

Challenges and accuracy loss

Observation and approach (dynamic two-D compression)

MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression Sheng Qiang Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [qiangsheng@sjtu.edu.cn](qiangsheng@sjtu.edu.cn) Ruiwei Chen Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [chenruiwei@sjtu.edu.cn](chenruiwei@sjtu.edu.cn) Yinpeng Wu Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China[wyp1536481268@foxmail.com](wyp1536481268@foxmail.com) arXiv :2607 .00760v 1 [ cs .LG] 1 Jul 2026 Jinyu Gu Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [gujinyu@sjtu.edu.cn](gujinyu@sjtu.edu.cn) Zhichao Hua∗ Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [zchua@sjtu.edu.cn](zchua@sjtu.edu.cn) Yubin Xia Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [xiayubin@sjtu.edu.cn](xiayubin@sjtu.edu.cn) Binyu Zang Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [byzang@sjtu.edu.cn](byzang@sjtu.edu.cn) Haibo Chen Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [haibochen@sjtu.edu.cn](haibochen@sjtu.edu.cn) Abstract Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the keyvalue (KV) cache a first-order serving cost. Because the cache grows linearly with context length, it can exhaust GPU memory, force smaller batches, and reduce serving throughput. Prior KV cache compression techniques typically target only the sequence dimension or only the channel dimension, which leaves limited headroom as context windows scale. Compressing both dimensions promises higher memory reduction, but applying the two forms of compression directly leads to significant accuracy loss. This paper introduces MosaicKV, a dynamic two-D (dimensional) KV cache compression system for extremely long-context serving. MosaicKV uses dynamic two-D compression to address the accuracy challenge, exploiting thenon-uniform importance distribution of elements within the KV cache. Instead of applying one compression pattern globally, MosaicKV identifies important elements for each KV vector and selects compression strategies at the granularity of KV cache segments. To address the performance challenge, where fine-grained sparsity and compression management overhead can offset the gains from compression, MosaicKV introduces compressed KV cache management. This mechanism uses underutilized GPU and CPU resources to maintain compressed KV caches and accelerate attention computation. Evaluation on an H800 GPU with multiple LLMs shows ∗ Corresponding author. that MosaicKV delivers up to 16× attention speedup, 4.8× lower decode latency, and 7.3× higher throughput than the uncompressed baseline. At the same time, it reduces memory usage by 3× and incurs only 1.76% average accuracy loss on LongBench and RULER. 1 Introduction Large language models (LLMs) are increasingly deployed in scenarios that demand extremely long contexts, such as repository-level code generation [56, 57], multi-document question answering [17, 45], and multi-turn agentic workflows [4, 8, 30, 46]. In response, frontier models [2, 3, 6, 7, 9] have rapidly extended their supported context windows to 1M tokens, e.g., GPT-5.4 [6] and Gemini 3.1 Pro [3] . However, long contexts impose a severe memory bottleneck through the key-value (KV) cache, whose size scales linearly with the sequence length during inference. For instance, serving 8 concurrent LLaMA-3.1-8B requests at 128K tokens requires 128 GB for the KV cache alone—already 8× the model weights; at 1M tokens, this grows to 1 TB, 64× the model weights. Such memory pressure exhausts GPU capacity, forces drastic reductions in batch size, and significantly increases per

FAQs

为什么长上下文 LLM 的 KV cache 会成为主要的服务成本？

KV cache 在推理时会随上下文长度线性增长，导致显存很快耗尽，从而降低可用 batch size并增大逐步解码延迟，最终降低吞吐。

MosaicKV 的关键创新是什么？

MosaicKV 引入动态的二维（two-D）KV cache 压缩，利用 KV 缓存中元素重要性分布不均的特点，在更细粒度的 KV 段上为每个 KV 向量选择合适的压缩策略。

二维压缩会带来怎样的精度问题，以及论文如何应对？

直接在 KV 的全流程中同时使用二维压缩会显著降低模型精度。MosaicKV通过动态两维压缩与“压缩 KV cache 管理”来控制精度损失，同时在计算阶段通过缓存管理加速注意力计算。

Jul 16, 2026

Upload by Anda