[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-273630-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-273630-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","asymmetric-attention-heads-structured-head-wise-context-allocation-for-transformer-attention-arxiv-260819203v1","Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention - arXiv 2608.19203v1","","Standard multi-head attention assigns every head an identical causal context span, even though different heads can play distinct roles in capturing local lexical/syntactic cues or longer-range dependencies. This work introduces Asymmetric Attention Heads (AAH), a head-wise context allocation framework that treats context length as an explicit per-head/per-group variable. AAH groups and organizes heads hierarchically, assigns discrete causal local windows, and preserves the standard Transformer flat output interface. Experiments on 4096-token seed-0 show lower validation loss for AAH-style local allocation variants; ablations highlight the importance of stable local allocation and head-window structure. Attention Coverage Ratio (ACR) is used as a routing diagnostic, not a FLOPs reduction claim.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/asymmetric-attention-heads-structured-head-wise-context-allocation-for-transformer-attention-arxiv-260819203v1/273630/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/asymmetric-attention-heads-structured-head-wise-context-allocation-for-transformer-attention-arxiv-260819203v1/273630.png","ImageObject",300,407,{"name":42,"@type":43},"Levi","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-06","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",14,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"How does Asymmetric Attention Heads (AAH) change standard multi-head attention?","Question",{"text":63,"@type":64},"AAH allows different heads (or head groups) to receive different-sized causal context windows. It controls per-head execution by selecting discrete causal local windows while keeping the standard flat Transformer interface for outputs.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What does Attention Coverage Ratio (ACR) measure in this work?",{"text":68,"@type":64},"ACR is used as a selected-window routing diagnostic and a proxy for window coverage. The paper separates ACR from hardware FLOPs measurement and does not treat it as a compute-reduction metric.",{"name":70,"@type":61,"acceptedAnswer":71},"Do the experiments show lower GPU FLOPs for AAH?",{"text":72,"@type":64},"No. The paper reports Nsight and FLOPs-lab diagnostics indicating the current implementations do not establish measured GPU-FLOPs savings, and it treats quality/routing effects as the primary claim.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},273630,1789472322,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":113,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":80,"read_time":144,"preview_page_count":4},7971461740909,"https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d","arXiv : 2608 . 19203v 1 [cs .CL] 10 Jun 2026  \nAsymmetric Attention Heads: Structured Head-Wise Context  \nAllocation  \nfor Transformer Attention  \nZimu Zhao [zimmer061310@gmail.com](zimmer061310@gmail.com)  \n§ [github.com/Zimmer061310/Asymmetric-Attention-Heads](github.com/Zimmer061310/Asymmetric-Attention-Heads)  \nAbstract  \nStandard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer-range relations such as entity interactions, discourse links, or state changes. We present Asymmetric Attention Heads (AAH), a head-wise contextallocation framework that treats context length as an explicit per-head or per-group allocation variable. AAH groups heads using feature-derived statistics, organizes these groups hierarchically, and assigns causal local windows while preserving the standard flat MHA output interface. In 4096-token seed-0 experiments, several AAH-style local-allocation variants achieve lower validation loss than pure full attention. Short-budget ablations show that stable local allocation and head-window assignment structure matter, while fixed/local controls can be competitive with adaptive hierarchy. We interpret AAH as a structured head-wise context-allocation mechanism for quality and analysis, with Attention Coverage Ratio (ACR) reported as a selected-window routing diagnostic.  \n(a) Standard MHA  \n\n|  | Input hidden states |  |  |\n| --- | --- | --- | --- |\n|  |  |  |  |\n| Shared Q/K/V projection |  |  |  |\n\n| \u003Cbr>Head  | Head |  | \u003Cbr>Head  | Head |\n| --- | --- | --- | --- | --- |\n\n|  |  | Full causal span\u003Cbr>for every head |  |  |\n| --- | --- | --- | --- | --- |\n|  |  |  |  |  |\n|  |  | Standard causal\u003Cbr>attention |  |  |\n|  |  |  |  |  |\n|  |  | Flat head concat |  |  |\n|  |  |  |  |  |\n|  | Output projection |  |  |  |\n|  |  |  |  |  |\n| Standard block output |  |  |  |  |\n\nUniform full-span execution  \nFigure 1: Standard MHA versus AAH-v3 . Standard MHA applies one full causal span to all heads before the flat output merge. AAH-v3 keeps the shared Q/K/V projection and flat Transformer interface, but adds a control branch that assigns per-head windows; the execution branch buckets heads by selected window and can use dense-masked, FlexAttention, or FlashAttention execution.  \n1 Introduction  \nTransformers rely on multi-head attention (MHA) [1], where each head is typically executed with a uniform attention pattern and full attended span. This design is simple and robust, but it also hidesa structural question: different heads may not need the same amount of context to make useful contributions. This motivates the central question in this paper: can a Transformer allocate context windows unevenly across heads while preserving, or sometimes improving, language-modeling quality?  \nWe use the term Asymmetric Attention Heads (AAH) for this design direction. Attention heads remain part of the standard MHA interface, but they are allowed to receive unequal context windows. AAH-v3 controls MHA execution across heads and head groups by selecting discrete causal local windows, while preserving the standard Transformer block interface and flat output merge. The method changes the head-wise context-allocation policy, rather than the output topology of the Transformer block.  \nOur evaluation scope is intentionally constrained. We treat Attention Coverage Ratio (ACR)1 as a selected-window coverage proxy, not a measured FLOPs metric. The dense-masked path does not establish skipped backend work; therefore the main rows omit backend work-accounting columns. True GPU FLOPs are measured only by Nsight-derived GPU floating-point-operation counters, and the current Nsight evidence does not support a FLOPs-reduction claim for the implementations in this paper.  \nAAH is backend-agnostic in the narrow sense that it chooses per-head context windows and then","cbCaimEjqitLVPKC","https://ap.wps.com/l/cbCaimEjqitLVPKC","pdf",527504,"English","# Abstract\n# 1 Introduction\n## Method and core design\n## Diagnostics and claim boundaries\n## Contributions overview","[{\"question\":\"How does Asymmetric Attention Heads (AAH) change standard multi-head attention?\",\"answer\":\"AAH allows different heads (or head groups) to receive different-sized causal context windows. It controls per-head execution by selecting discrete causal local windows while keeping the standard flat Transformer interface for outputs.\"},{\"question\":\"What does Attention Coverage Ratio (ACR) measure in this work?\",\"answer\":\"ACR is used as a selected-window routing diagnostic and a proxy for window coverage. The paper separates ACR from hardware FLOPs measurement and does not treat it as a compute-reduction metric.\"},{\"question\":\"Do the experiments show lower GPU FLOPs for AAH?\",\"answer\":\"No. The paper reports Nsight and FLOPs-lab diagnostics indicating the current implementations do not establish measured GPU-FLOPs savings, and it treats quality/routing effects as the primary claim.\"}]","Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention - arXiv 2608.19203v1 | PDF",76]