[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82559-en":3,"doc-seo-82559-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82559,962075006959,"Anda","https://ap-avatar.wpscdn.com/avatar/e0002397efbe92a78e?_k=1776741047341049297",8,"Research & Report","MosaicKV Serving Long-Context LLM with Dynamic Two-D KV Cache Compression","Long-context LLM services sustain prompts with hundreds of thousands to millions of tokens, making the KV cache a dominant serving cost. Since cache size scales linearly with context length, GPU memory is quickly exhausted, batches shrink, and throughput drops. Prior KV compression often targets only sequence or only channel, limiting gains at larger windows. MosaicKV proposes dynamic two-dimensional KV compression that selects strategies per KV vector segment using non-uniform element importance, and manages compressed caches across GPU/CPU. Experiments on an H800 show up to 16× attention speedup, 3× lower memory use, and only 1.76% average accuracy loss.","MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression  \nSheng Qiang  \nInstitute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [qiangsheng@sjtu.edu.cn](qiangsheng@sjtu.edu.cn)  \nRuiwei Chen  \nInstitute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [chenruiwei@sjtu.edu.cn](chenruiwei@sjtu.edu.cn)  \nYinpeng Wu  \nInstitute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China[wyp1536481268@foxmail.com](wyp1536481268@foxmail.com)  \narXiv :2607 .00760v 1 [ cs .LG] 1 Jul 2026  \nJinyu Gu  \nInstitute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [gujinyu@sjtu.edu.cn](gujinyu@sjtu.edu.cn)  \nZhichao Hua∗ Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [zchua@sjtu.edu.cn](zchua@sjtu.edu.cn)  \nYubin Xia  \nInstitute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [xiayubin@sjtu.edu.cn](xiayubin@sjtu.edu.cn)  \nBinyu Zang  \nInstitute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [byzang@sjtu.edu.cn](byzang@sjtu.edu.cn)  \nHaibo Chen  \nInstitute of Parallel and Distributed Systems, Shanghai Jiao Tong University Shanghai, China [haibochen@sjtu.edu.cn](haibochen@sjtu.edu.cn)  \nAbstract  \nLong-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the keyvalue (KV) cache a first-order serving cost. Because the cache grows linearly with context length, it can exhaust GPU memory, force smaller batches, and reduce serving throughput. Prior KV cache compression techniques typically target only the sequence dimension or only the channel dimension, which leaves limited headroom as context windows scale. Compressing both dimensions promises higher memory reduction, but applying the two forms of compression directly leads to significant accuracy loss.  \nThis paper introduces MosaicKV, a dynamic two-D (dimensional) KV cache compression system for extremely long-context serving. MosaicKV uses dynamic two-D compression to address the accuracy challenge, exploiting thenon-uniform importance distribution of elements within the KV cache. Instead of applying one compression pattern globally, MosaicKV identifies important elements for each KV vector and selects compression strategies at the granularity of KV cache segments. To address the performance challenge, where fine-grained sparsity and compression management overhead can offset the gains from compression, MosaicKV introduces compressed KV cache management. This mechanism uses underutilized GPU and CPU resources to maintain compressed KV caches and accelerate attention computation. Evaluation on an H800 GPU with multiple LLMs shows  \n∗ Corresponding author.  \nthat MosaicKV delivers up to 16× attention speedup, 4.8× lower decode latency, and 7.3× higher throughput than the uncompressed baseline. At the same time, it reduces memory usage by 3× and incurs only 1.76% average accuracy loss on LongBench and RULER.  \n1 Introduction  \nLarge language models (LLMs) are increasingly deployed in scenarios that demand extremely long contexts, such as repository-level code generation [56, 57], multi-document question answering [17, 45], and multi-turn agentic workflows [4, 8, 30, 46]. In response, frontier models [2, 3, 6, 7, 9] have rapidly extended their supported context windows to 1M tokens, e.g., GPT-5.4 [6] and Gemini 3.1 Pro [3] .  \nHowever, long contexts impose a severe memory bottleneck through the key-value (KV) cache, whose size scales linearly with the sequence length during inference. For instance, serving 8 concurrent LLaMA-3.1-8B requests at 128K tokens requires 128 GB for the KV cache alone—already 8× the model weights; at 1M tokens, this grows to 1 TB, 64× the model weights. Such memory pressure exhausts GPU capacity, forces drastic reductions in batch size, and significantly increases per","cbCaiscTjTDN0vKt","https://ap.wps.com/l/cbCaiscTjTDN0vKt","pdf",3814457,6,1,15,"English","en",105,"# Abstract\n# Introduction\n## Long-context LLM and KV-cache bottleneck\n## Prior KV-cache compression approaches\n## Challenges and accuracy loss\n## Observation and approach (dynamic two-D compression)","[{\"question\":\"为什么长上下文 LLM 的 KV cache 会成为主要的服务成本？\",\"answer\":\"KV cache 在推理时会随上下文长度线性增长，导致显存很快耗尽，从而降低可用 batch size并增大逐步解码延迟，最终降低吞吐。\"},{\"question\":\"MosaicKV 的关键创新是什么？\",\"answer\":\"MosaicKV 引入动态的二维（two-D）KV cache 压缩，利用 KV 缓存中元素重要性分布不均的特点，在更细粒度的 KV 段上为每个 KV 向量选择合适的压缩策略。\"},{\"question\":\"二维压缩会带来怎样的精度问题，以及论文如何应对？\",\"answer\":\"直接在 KV 的全流程中同时使用二维压缩会显著降低模型精度。MosaicKV通过动态两维压缩与“压缩 KV cache 管理”来控制精度损失，同时在计算阶段通过缓存管理加速注意力计算。\"}]",1784181525,38,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"mosaickv-serving-long-context-llm-with-dynamic-two-d-kv-cache-compression","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/mosaickv-serving-long-context-llm-with-dynamic-two-d-kv-cache-compression/82559/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"为什么长上下文 LLM 的 KV cache 会成为主要的服务成本？","Question",{"text":76,"@type":77},"KV cache 在推理时会随上下文长度线性增长，导致显存很快耗尽，从而降低可用 batch size并增大逐步解码延迟，最终降低吞吐。","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"MosaicKV 的关键创新是什么？",{"text":81,"@type":77},"MosaicKV 引入动态的二维（two-D）KV cache 压缩，利用 KV 缓存中元素重要性分布不均的特点，在更细粒度的 KV 段上为每个 KV 向量选择合适的压缩策略。",{"name":83,"@type":74,"acceptedAnswer":84},"二维压缩会带来怎样的精度问题，以及论文如何应对？",{"text":85,"@type":77},"直接在 KV 的全流程中同时使用二维压缩会显著降低模型精度。MosaicKV通过动态两维压缩与“压缩 KV cache 管理”来控制精度损失，同时在计算阶段通过缓存管理加速注意力计算。","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]