[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82164-en":3,"doc-seo-82164-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82164,687197207057,"Sage","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","COBS Cumulant Order Block Sparse Attention","Block sparse attention offers a hardware-friendly way to reduce the KV cache read bottleneck in large language models, yet it is uncommon in leading open-weight LLMs. The study analyzes DeepSeek’s Native Sparse Attention (NSA) to isolate block selection, showing that quality matches dense attention when the selected blocks maximize attention mass. Exact mass requires reading all keys, motivating compact approximations. Using a cumulant expansion, the work explains why first-order estimators fail and introduces COBS, which stores a compressed second-order statistic per block and substantially closes the dense gap on 32k RULER retrieval.","arXiv :2607 .09052v 1 [ cs .LG] 10 Jul 2026  \nCOBS: CUMULANT ORDER BLOCK SPARSE ATTENTION  \nAlexander Tian* Aditya Ghai* Sanjit Neelam Zaal Vasania Akshay Mishra  \nMatX  \n{alexander,adi,sanjit,zaal,[akshay}@matx.com](akshay}@matx.com)  \nABSTRACT  \nBlock sparse attention is a hardware friendly way to alleviate the key–value (KV) cache read bottleneck in large language models (LLMs) . However, it is not prevalent among leading openweight LLMs, which rely instead on dense attention or fine-grained selection, thereby motivating our analysis. We study DeepSeek’s Native Sparse Attention (NSA) [1] as a representative method, whose three-branch design lets us isolate block selection, the most challenging and consequential stage. We formalize selection and reduce it to ranking blocks by a single quantity, the attention mass: the sum of a block’s attention scores. We show that if selection retrieves the blocks with the largest attention mass, block sparse attention can match the quality of dense attention. However, computing the exact attention mass requires reading every key, so the problem of block selection ultimately reduces to approximating this mass from a compact summary instead of the full keys.  \nVia a cumulant expansion, we show why existing methods falter: their selection strategies attempt to estimate the attention mass, but are confined to a first-order approximation. Therefore, we propose COBS (Cumulant Order Block Sparse Attention), an attention method that builds on NSA, incorporating a novel selector that stores a compressed second-order statistic per block. On the 32k RULER long-context retrieval benchmark [2], COBS raises the NSA baseline’s meanscore from 0.2999 to 0.8195, approaching dense attention at 0.9040 and closing about 86% of the gap, while using only 1.21 × the KV cache read traffic of the NSA baseline and 15.15 × less read traffic than dense. The same model preserves short-context behavior and attains lower position-wise negative log-likelihood (NLL) than dense attention in our comparison.  \n1 Introduction  \nTransformer inference at long context is bottlenecked by reading the key–value (KV) cache: at each decode step, attention reads the keys and values of every past token, so decoding is limited by memory bandwidth rather than compute and leaves hardware underutilized. Sparse attention reduces this cost by reading the keys and values of only a fraction of past tokens.  \nSparse attention methods differ in what they keep and how they choose it. Fixed-pattern methods attend to predetermined local windows and global tokens (Longformer [3], BigBird [4]); KV-eviction methods drop low-importance tokens (StreamingLLM [5], H2O [6], SnapKV [7]); and low-rank methods compress the keys themselves [8] . Closest to our setting are query-aware block selectors, which summarize contiguous blocks of the cache offline and, per query, run fine-grained attention over the highest-scoring blocks; this family includes Quest [9] and the learned block selector of Native Sparse Attention (NSA) [1] . We return to these methods in Section 6 .  \nAmong these families, block sparsity is arguably the most hardware friendly: attending to contiguous blocks yields contiguous memory accesses, and a coarse block granularity requires a far smaller top-k than selecting individual tokens. Despite this affinity for hardware, block sparsity is largely absent from leading open-weight LLMs. Recent releases instead span the alternatives: dense grouped-query attention (GQA [10]; e.g. Hunyuan Hy3 [11]), latent KV compression (Multi-head Latent Attention (MLA) [12], used by Kimi K2.6 [13]), local windows interleaved with periodic global layers (Gemma 4 [14]), and fine-grained token selection (DeepSeek Sparse Attention (DSA) [15], adopted in modified form by GLM-5.2 [16]) . Even the closest case, DeepSeek-V4’s CSA [17], selects over blocks only four tokens wide and pairs them with a compression branch (HCA) that performs no selection at all. Large-block s","cbCait17Pw46srFM","https://ap.wps.com/l/cbCait17Pw46srFM","pdf",716208,1,17,"English","en",105,"# Introduction\n## Long-context KV cache bottleneck\n## Related sparse attention families\n# 2k RULER\n## Accuracy vs KV-read traffic","[{\"question\":\"What is the main motivation for COBS and block sparse attention?\",\"answer\":\"Long-context transformer inference is constrained by KV cache reads at each decoding step. Block sparse attention reduces this by reading keys and values for only a subset of past tokens, lowering memory bandwidth demand.\"},{\"question\":\"How does the paper characterize block selection in NSA?\",\"answer\":\"Block selection can be formalized as ranking cache blocks by a single quantity called attention mass, defined as the sum of a block’s attention scores.\"},{\"question\":\"Why do existing block selection methods underperform, and what does COBS change?\",\"answer\":\"Existing methods attempt to estimate attention mass using first-order approximations, which the cumulant expansion shows are insufficient. COBS modifies NSA with a new selector that stores a compressed second-order statistic per block to improve mass estimation and recover dense-like quality.\"}]",1784178537,43,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"cobs-cumulant-order-block-sparse-attention","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/cobs-cumulant-order-block-sparse-attention/82164/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What is the main motivation for COBS and block sparse attention?","Question",{"text":74,"@type":75},"Long-context transformer inference is constrained by KV cache reads at each decoding step. Block sparse attention reduces this by reading keys and values for only a subset of past tokens, lowering memory bandwidth demand.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does the paper characterize block selection in NSA?",{"text":79,"@type":75},"Block selection can be formalized as ranking cache blocks by a single quantity called attention mass, defined as the sum of a block’s attention scores.",{"name":81,"@type":72,"acceptedAnswer":82},"Why do existing block selection methods underperform, and what does COBS change?",{"text":83,"@type":75},"Existing methods attempt to estimate attention mass using first-order approximations, which the cumulant expansion shows are insufficient. COBS modifies NSA with a new selector that stores a compressed second-order statistic per block to improve mass estimation and recover dense-like quality.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]