[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83948-en":3,"doc-seo-83948-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83948,687197207639,"Asher","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Benchmarking KV Cache Optimizations for Long-Context Serving","Long-context large language model serving is constrained by KV-cache growth, and existing compression techniques are hard to compare because they are tested on different models, tasks, budgets, and serving stacks. The study introduces a workload-aware benchmark covering quantization, pruning, and merging methods (KIVI, TurboQuant, SnapKV, CaM) on LongBench-style QA, few-shot learning, and summarization using Llama-3.1-8B and Mistral-7B. It evaluates task quality and system metrics like throughput and time-to-first-token across context-length buckets. Compression ratio alone poorly predicts end-to-end performance; results guide workload-specific deployment choices.","Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving [Experiment,  \nAnalysis & Benchmark]  \nNikita Agrawal  \nUniversity of Bayreuth Bayreuth, Germany [Nikita.Agrawal@uni-bayreuth.de](Nikita.Agrawal@uni-bayreuth.de)  \nRuben Mayer  \nUniversity of Bayreuth Bayreuth, Germany [Ruben.Mayer@uni-bayreuth.de](Ruben.Mayer@uni-bayreuth.de)  \narXiv :2607 .05399v1 [ cs .CL] 3 May 2026  \nABSTRACT  \nLarge language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks. This paper presents a workload-aware benchmark of representative KVcache optimization mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM, evaluated on LongBench-style multi-document QA, single-document QA, few-shot learning, and summarization workloads using Llama-3.1- 8B-Instruct and Mistral-7B-Instruct-v0 .3. The benchmark measures task quality, mean output throughput, mean time-to-first-token, and realized compression ratio across context-length buckets. The results show that the compression ratio alone is a poor predictor of end-to-end performance. KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio. These findings motivate workload-aware selection of KV-cache mechanisms rather than one-size-fits-all compression and provide deployment guidance for long-context serving systems.  \nPVLDB Reference Format:  \nNikita Agrawal and Ruben Mayer. Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving [Experiment, Analysis & Benchmark] . PVLDB, 14(1): XXX-XXX, 2020 .  \ndoi:XX.XX/XXX.XX  \nPVLDB Artifact Availability:  \nThe source code, data, and/or other artifacts have been made available at [https://github.com/nikagrwal/Benchmarking-KV-Cache-Optimizations](https://github.com/nikagrwal/Benchmarking-KV-Cache-Optimizations)across-Task-Quality-and-System-Performance.  \n1 INTRODUCTION  \nThe adoption of large language models (LLMs) has grown rapidly across a wide range of data-intensive applications, including document summarization, multi-turn dialogue, and code analysis [3, 13] .  \nThis work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit [https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of](https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of)[ ](https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of)[this license. For any use beyond those covered by this license](this license. For any use beyond those covered by this license), [obtain permission by](obtain permission by)[emailing info@vldb.org. Copyright](emailing info@vldb.org. Copyright) is held by the owner/author(s). Publication rights licensed to the VLDB Endowment.  \nProceedings of the VLDB Endowment, Vol. 14, No. 1 ISSN 2150-8097 .  \ndoi:XX.XX/XXX.XX  \nAs these applications increasingly rely on long-context inputs, efficient LLM serving has emerged as a critical systems challenge. In particular, the Key–Value (KV) cache used in LLMs grows linearly with input length, leading to substantial memory and bandwidth overhead during inference [31]. This has motivated a surge of recent work on KV cache optimization techniques, such as quantization, pruning, and merging, to enable scalable long-context serving.  \nThe data management community has recently begun to address these challenges by rethinking LLM serving stacks and system-level optimizations. Previous works have explored enhancing LLM serving by improving efficiency in inference pipelines, memory-aware scheduling, and hardware-conscious optimizations [25, 28, 35, 44] . A gro","cbCaibr65ng89Byv","https://ap.wps.com/l/cbCaibr65ng89Byv","pdf",1014152,4,1,13,"English","en",105,"# Abstract\n# Introduction\n## Motivation: KV-cache growth in long-context workloads\n## Gap: lack of unified, workload-aware evaluation\n## Contribution: workload-aware benchmark and metrics","[{\"question\":\"Why is KV-cache optimization important for long-context LLM serving?\",\"answer\":\"KV-cache size grows linearly with input length, creating major memory and bandwidth overhead during inference. This motivates techniques such as quantization, pruning, and merging to enable scalable long-context serving.\"},{\"question\":\"Which KV-cache optimization methods are evaluated in the benchmark?\",\"answer\":\"The benchmark covers representative mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM.\"},{\"question\":\"How do the results explain the relationship between compression ratio and end-to-end performance?\",\"answer\":\"The findings show that compression ratio alone is a poor predictor of end-to-end performance, since different methods exhibit distinct trade-offs across task quality and system efficiency.\"}]",1784191616,33,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"benchmarking-kv-cache-optimizations-for-long-context-serving","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/benchmarking-kv-cache-optimizations-for-long-context-serving/83948/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is KV-cache optimization important for long-context LLM serving?","Question",{"text":75,"@type":76},"KV-cache size grows linearly with input length, creating major memory and bandwidth overhead during inference. This motivates techniques such as quantization, pruning, and merging to enable scalable long-context serving.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Which KV-cache optimization methods are evaluated in the benchmark?",{"text":80,"@type":76},"The benchmark covers representative mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM.",{"name":82,"@type":73,"acceptedAnswer":83},"How do the results explain the relationship between compression ratio and end-to-end performance?",{"text":84,"@type":76},"The findings show that compression ratio alone is a poor predictor of end-to-end performance, since different methods exhibit distinct trade-offs across task quality and system efficiency.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]