[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-273677-105":59,"doc-detail-273677-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","learning-how-to-forget-fine-tuning-for-long-context-sparse-attention-abstract","Learning how to Forget - Fine-tuning for Long-Context Sparse Attention - Abstract","","A lot of prior work tackled key-value (KV) cache selection and compression using sparse attention to enable long-context inference for transformer language models under limited hardware budgets. This work introduces a fine-tuning method for sparse attention that applies to any KV cache policy, runs on moderate resources (e.g., one Nvidia A100 GPU with 40GB RAM), and co-adapts the model with the policy, frequently outperforming exact-attention training. It also provides an efficient H2O sparse attention implementation with SDPA kernel support and releases KeysAndValues for long-context inference and fine-tuning.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":30,"@type":76,"position":81},"https://docshare.wps.com/document/technology/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/learning-how-to-forget-fine-tuning-for-long-context-sparse-attention-abstract/273677/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/learning-how-to-forget-fine-tuning-for-long-context-sparse-attention-abstract/273677.png","ImageObject",300,407,{"name":92,"@type":93},"River Wang","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-19","2026-09-15",true,{"@type":102,"interactionType":103,"userInteractionCount":44},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What problem does this paper address?","Question",{"text":112,"@type":113},"It addresses how to post-train a transformer language model with sparse attention on a moderate hardware budget while keeping the method general for practical deployment.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"What is the proposed fine-tuning method’s key capability?",{"text":117,"@type":113},"The method works for any KV cache policy and lets the model co-adapt with the policy, often outperforming models trained with exact attention.",{"name":119,"@type":110,"acceptedAnswer":120},"How is H2O (heavy-hitter oracle) improved in the paper?",{"text":121,"@type":113},"The paper demonstrates improvements to H2O, including an efficient implementation with dedicated scaled dot product attention (SDPA) kernel support, plus code to return summed attention weights alongside an SDPA kernel.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},273677,1790377447,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":29,"category_name":30,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":44,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145},1099514067438,"https://ap-avatar.wpscdn.com/avatar/100002539ee87300030?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780474512215547542","Learning how to Forget: Fine-tuning for Long-Context Sparse  \nAttention  \narXiv : 2608 . 19920v 1 [cs .CL] 20 Aug 2026  \nMatthias Seeger∗ Amazon Web Services [mseeger@gmail. com](mseeger@gmail. com)  \nZeyu Zhang University of Amsterdam  \n[z. zhang2@uva. nl](z. zhang2@uva. nl)  \nVihang Patil Amazon  \n[pvihang@amazon. de](pvihang@amazon. de)  \nKonstantinos Benidis Amazon Web Services [kbenidis@amazon. de](kbenidis@amazon. de)  \nSebastian Schelter Technical University Berlin [schelter@tu-berlin. de](schelter@tu-berlin. de)  \nAugust 21, 2026  \nAbstract  \nA lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for fine-tuning models with sparse attention. It works for any KV cache policy, runs on a moderate hardware budget (e.g., a single Nvidia A100 GPU with 40 GB RAM), and allows the model to co-adapt with the policy, often outperforming models trained with exact attention (sequence parallelism) . We also provide an efficient implementation of H2O sparse attention (the leading policy in our experiments) with dedicated scaled dot product attention kernel support. KeysAndValues, a new open source library for long-context inference and fine-tuning, provides easy-to-use and performant code for all methods discussed here.  \n1 Introduction  \nModern large language models need to process very long contexts (i.e., number of tokens) for calling many tools with sizable outputs [47, 16], running chain of thought reasoning [62], or sustaining multi-turn conversations. While naive transformer implementations scale quadratically in compute and linearly in memory with context width, a lot of progress has been made on approximations with essentially linear time and constant memory scaling. A particularly fruitful direction is sparse attention, where key-value (KV) information is stored in a fixed-size KV cache, slots of which are evicted once it is full, and many different eviction policies have been proposed.  \nIn this paper, we address the problem of how to post-train a transformer language model with sparse attention on a moderate hardware budget (our experiments are run computing gradients for a 4B weights model on a single 1 Nvidia A100 GPU with 40 GB RAM) .  \n∗ Correspondence [to](to mseeger@gmail.com)[ mseeger@gmail.com](to mseeger@gmail.com)  \n1We use 4 GPUs with distributed data parallel training to attain a larger batch size.  \nOur novel method works for any KV cache policy and requires no further approximations beyond sparse attention. As we demonstrate in experiments on a range of long-context benchmarks, our training algorithm allows the model to co-adapt with the KV cache policy, often outperforming models trained with exact attention (sequence parallelism) . Moreover, our fine-tuning method runs on resources comparable to sparse attention inference. It can be combined with orthogonal KV cache compression strategies such as grouped query attention [1] or quantization [27, 40] .  \nThe heavy-hitter oracle (H2O) [72] is one of the most prominent sparse attention policies. We demonstrate several improvements to H2O, leading to a much more efficient implementation with dedicated scaled dot product attention (SDPA) kernel support. Variants of H2O outperform other KV cache policies in our experiments, and our fast implementation takes a big step towards latencies competitive with SotA inference libraries such as vLLM [30], which use context or sequence parallelism almost exclusively. In summary, our contributions are:  \n• New method for fine-tuning transformer language models with sparse attention and arbitrary KV cache policy in place. This method runs on resources comparable to sparse attention inference. It combines nested activation checkpointing and CPU offloading with exploiting a linear KV cache buffer recurrence by way of autograd saved tensors packing. Our method can ","cbCaildEe6YJOWGq","https://ap.wps.com/l/cbCaildEe6YJOWGq","pdf",536910,39,"English","# Introduction\n## Problem: post-training with sparse attention\n## Approach: co-adaptation for arbitrary KV cache policies\n# Related Work\n## KV cache compression and sparse attention variants\n## Heavy-hitter oracle and related methods","[{\"question\":\"What problem does this paper address?\",\"answer\":\"It addresses how to post-train a transformer language model with sparse attention on a moderate hardware budget while keeping the method general for practical deployment.\"},{\"question\":\"What is the proposed fine-tuning method’s key capability?\",\"answer\":\"The method works for any KV cache policy and lets the model co-adapt with the policy, often outperforming models trained with exact attention.\"},{\"question\":\"How is H2O (heavy-hitter oracle) improved in the paper?\",\"answer\":\"The paper demonstrates improvements to H2O, including an efficient implementation with dedicated scaled dot product attention (SDPA) kernel support, plus code to return summed attention weights alongside an SDPA kernel.\"}]","Learning how to Forget - Fine-tuning for Long-Context Sparse Attention - Abstract | PDF",1789472460,98]