[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-327133-105":59,"doc-detail-327133-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","kara-sliding-window-kv-cache-compression-for-efficient-serving-of-reasoning-llms-under-memory-constraints","Kara - Sliding-Window KV Cache Compression for Efficient Serving of Reasoning LLMs under Memory Constraints","","Reasoning language models often produce long chain-of-thought traces, which causes KV cache to grow rapidly during decoding, increasing memory usage, latency, and limiting output throughput—especially in memory-constrained concurrent serving. Existing KV compression methods suffer from threshold-triggered throughput issues and rigid or fixed-boundary retention, leading to significant information loss. Kara introduces sliding-window, decoding-time compression with bidirectional attention scoring and Token2Chunk expansion, and KvLLM built on vLLM to reduce KV memory while improving throughput.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/kara-sliding-window-kv-cache-compression-for-efficient-serving-of-reasoning-llms-under-memory-constraints/327133/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/kara-sliding-window-kv-cache-compression-for-efficient-serving-of-reasoning-llms-under-memory-constraints/327133.png","ImageObject",300,407,{"name":92,"@type":93},"Lucas Martin","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-07","2026-09-21",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",16,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"Why does KV cache become a bottleneck for reasoning LLMs during decoding?","Question",{"text":113,"@type":114},"Reasoning models generate long chain-of-thought traces, so KV cache accumulates quickly in the decoding phase. This increases memory overhead and decoding latency, which reduces output throughput in memory-constrained concurrent serving.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"What limitations are identified in existing KV cache compression methods?",{"text":118,"@type":114},"The paper highlights two limitations: threshold-triggered compression can lower throughput under concurrency, and many methods retain rigid fixed chunks or isolated KV pairs, which can worsen information loss. It also fails to preserve flexible-sized chunks at arbitrary token positions.",{"name":120,"@type":111,"acceptedAnswer":121},"How does Kara improve KV cache compression compared with prior work?",{"text":122,"@type":114},"Kara performs decoding-time compression only on the recently generated context using a sliding window. It scores and selects informative KV pairs with bidirectional attention, then uses Token2Chunk to flexibly preserve important semantic information by expanding selected KV pairs into chunks.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},327133,1790143668,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":145,"read_time":146,"preview_page_count":4},8796095360427,"https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d","Kara: Sliding-Window KV Cache Compression for Efficient Serving of Reasoning LLMs under Memory  \nConstraints  \narXiv :2607 .0 1237v2 [ cs .CL] 3 Jul 2026  \nHan Shen∗ [kingsutherszsz48@gmail.com](kingsutherszsz48@gmail.com)  \nYuyang Wu∗  \nCarnegie Mellon University [yuyangwu@cs.cmu.edu](yuyangwu@cs.cmu.edu)  \nJunpu Yu  \nCarnegie Mellon University  \nOlexandr Isayev†  \nCarnegie Mellon University  \nAbstract  \nReasoning language models often generate long chain-of-thought (CoT), which accumulates a massive KV cache during the decoding phase and incurs high decoding latency and limited output throughput. To address these issues, KV cache compression has emerged as a promising technique for reducing memory footprint by selectively removing unimportant KV pairs while preserving useful ones for subsequent decoding. Nevertheless, we identify two key limitations in existing KV cache compression methods: 1) Their threshold-triggered compression policy may reduce output throughput under memory-constrained concurrent serving, and may fully eliminate KV pairs from certain blocks of the sequence, potentially worsening information loss. 2) They typically retain either isolated KV pairs or fixed-size chunks with rigid boundaries, failing to preserve flexible-sized chunks at arbitrary token positions. To overcome these limitations, we propose Kara, a slidingwindow KV cache compression method that performs decoding-time compression by operating only on the recently generated context. Kara leverages bidirectional attention to score and select informative KV pairs in the window. To enable flexible preservation of important semantic information, we design a Token2Chunk module to expand a subset of selected KV pairs into chunks. Furthermore, we adapt Kara to PagedAttention and develop KvLLM, an inference framework built upon vLLM, which reduces KV cache memory usage and improves output throughputin memory-constrained environments. Kara preserves nearly 100% of the fullKV accuracy while retaining only 20% of the KV cache, and KvLLM improves throughput by 12.75% on average over vanilla vLLM under memory-constrained concurrent serving.  \n1 Introduction  \nChain-of-thought reasoning [1, 2, 3, 4] has endowed LLMs with powerful capabilities for solving complex tasks. However, reasoning models often generate long traces, which lead to a substantial growth of KV cache during decoding and impose significant memory overhead [5, 6] . For example, caching 128K tokens in Qwen3-14B with FP16 precision requires approximately 20 GB of memory [3] . This overhead becomes especially pronounced in memory-constrained concurrent serving settings, where the rapidly growing KV cache increases decoding latency and causes input requests to wait for available memory resources, thereby reducing overall output throughput.  \n∗ Han Shen and Yuyang Wu are co–first authors and contribute equally to this paper.†Corresponding Author  \nPreprint.  \n vLLM  vLLM with SnapKV  \nThroughput  \n3000  \n2000  \n1000  \n0 1024 2048 3072 4096  \nDecoding Step  \n(b)  \nFigure 1: (a) Average output throughput under different batch sizes (i.e., the predefined maximum number of decoding sequences) . (b) The actual number of decoding sequences varies with decoding steps. The decoding step denotes the number of global decoding iterations. The predefined maximum number of concurrently decoding sequences is set to 256 . SnapKV constrains the KV cache size, freeing memory to decode more sequences concurrently.  \nCompression Count  \nFigure 2: Remaining KV pairs under different compression counts. We track the remaining KV pairs of tokens from different regions for a certain attention head across repeated compression steps. We observe that the semantic information of certain regions is completely eliminated as compression proceeds.  \nTo address these issues, KV cache compression [7, 8, 9], an inference-time approach, has garnered increasing attention. By selectively retaining important KV pairs, KV cache compressi","cbCaivw3UBr5cxEv","https://ap.wps.com/l/cbCaivw3UBr5cxEv","pdf",660859,18,"English","# Abstract\n# 1 Introduction\n## Chain-of-thought reasoning and KV cache growth\n## KV cache compression approaches and limitations","[{\"question\":\"Why does KV cache become a bottleneck for reasoning LLMs during decoding?\",\"answer\":\"Reasoning models generate long chain-of-thought traces, so KV cache accumulates quickly in the decoding phase. This increases memory overhead and decoding latency, which reduces output throughput in memory-constrained concurrent serving.\"},{\"question\":\"What limitations are identified in existing KV cache compression methods?\",\"answer\":\"The paper highlights two limitations: threshold-triggered compression can lower throughput under concurrency, and many methods retain rigid fixed chunks or isolated KV pairs, which can worsen information loss. It also fails to preserve flexible-sized chunks at arbitrary token positions.\"},{\"question\":\"How does Kara improve KV cache compression compared with prior work?\",\"answer\":\"Kara performs decoding-time compression only on the recently generated context using a sliding window. It scores and selects informative KV pairs with bidirectional attention, then uses Token2Chunk to flexibly preserve important semantic information by expanding selected KV pairs into chunks.\"}]","Kara - Sliding-Window KV Cache Compression for Efficient Serving of Reasoning LLMs under Memory Constraints | PDF",1789976502,45]