[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85678-en":3,"doc-seo-85678-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85678,549758252649,"Ivy","https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819",8,"Research & Report","Low-Rank Attention Residuals","Attention Residuals (ATTNRES) modify Transformer residual connections by using depth-wise attention over earlier sub-layer outputs, reusing each output as both routing key and mixed value. This couples routing to representation and makes routing scores scale with the hidden width, increasing representational tension and compute pressure. Low-Rank Attention Residuals (LR-ATTNRES) keep full-dimensional residual values while computing depth-routing keys in a smaller r-dimensional space. Projected and sliced variants decouple or simplify routing, improving validation loss while reducing added FLOPs.","arXiv :2607 .09694v1 [ cs .LG] 19 Jun 2026  \nLOW-RANK ATTENTION RESIDUALS  \nJonathan Su  \nIndependent Researcher  \n[270985@learning.gsis.edu.hk](270985@learning.gsis.edu.hk)  \nABSTRACT  \nAttention Residuals (ATTNRES) replace the fixed residual sum with depth-wise attention over previous sub-layer outputs in Large Language Models (LLMs), but use each output as both a full-dimensional key and value. This couples routing with representation and makes depth-routing scores scale with hidden width d. We propose Low-Rank Attention Residuals (LR-ATTNRES), which keep fulldimensional residual values while using r-dimensional keys, with r ≪ d, for routing. Projected LR-AttnRes (P-LR-ATTNRES) emits learned low-rank keys from existing output projections, decoupling routing from residual content and achieving the best validation loss among the variants tested. Sliced LR-AttnRes (S-LR-ATTNRES) uses the last r dimensions of each value as the routing key, removing the auxiliary key-projection path and reducing total residual-side FLOPs while still improving performance. Comprehensive sweeps show that depth-wise routing can be effective with far fewer dimensions than the model width. We release code and models to facilitate future research.  \nVa l idat ion Loss  \n3.000  \n2.985  \n2.970  \n2.955  \n2.940  \nBaseline  \nn=4  \nn=8  \nn=16  \nn=4 r=64  \nn=16 r=64  \nn=8 r=64  \nBaseline Transformer Standard AttnRes Sliced LR-AttnRes Projected LR-AttnRes  \nFull  \nFull  \nr=16,32,64,128,256,512  \nFull r=16  \nn=16 r=32  \nn=4 r=32 n=8 r=32  \nFull r=32  \nFull r=64  \n0.0 0.5 1.0 1.5 2.0 2.5  \nAdded FLOPs (% of Baseline Transformer)  \nFigure 1: Validation loss versus percentage of added FLOPs relative to the non-embedding Transformer core for the baseline, ATTNRES, and low-rank ATTNRES variants near 0.5B parameters trained on 10B tokens. For projected low-rank variants, the plotted FLOPs include the auxiliary key-projection path; for standard ATTNRES and sliced low-rank variants, they are the depth-wise residual kernel FLOPs.  \n1 INTRODUCTION  \nResidual connections are central to the optimization of deep networks (He et al., 2016) . This is especially true in Transformer-based large language models (Vaswani et al., 2017), where residual streams allow information to accumulate across depth and also serve as gradient highways for optimization. In a PreNorm decoder (Xiong et al., 2020), each attention or feed-forward sub-layer reads a normalized version of the current stream and writes an output back into it. This fixed additive rule is simple and stable, but it gives every previous output a fixed unit coefficient. The model can  \nlearn what each sub-layer writes, yet it cannot directly choose which previous writes should be emphasized when constructing the next hidden state.  \nAttention Residuals (ATTNRES) address this limitation by replacing fixed accumulation with attention over depth (Team et al., 2026) . At each residual site, the hidden state is formed as a learned weighted combination of earlier sub-layer outputs. This is analogous to token attention, but the “tokens” are previous sub-layer outputs. The full version attends over all previous sub-layer outputs, while BLOCK ATTNRES groups several outputs into block summaries to reduce storage and communication. The standard ATTNRES design makes the sub-layer output oi ∈ Rd serve two roles at once. It is the value mixed into the residual stream, and after normalization it is also the key used to decide how strongly that source should be selected. These roles need not require the same representation. A residual output should remain a full-capacity carrier of information for downstream layers. A routing key only needs to distinguish among depth sources. Using oi as both value and key therefore creates a representational tension, and it also makes the score computation scale with the hidden dimension d despite depth attention selecting among only tens of sub-layer sources rather than the vastly larger token axis for ","cbCaivI52fPXwWfH","https://ap.wps.com/l/cbCaivI52fPXwWfH","pdf",397546,1,16,"English","en",105,"# Abstract\n# Introduction\n## Residual connections in Transformers\n## Attention Residuals (ATTNRES)\n## Low-Rank Attention Residuals (LR-ATTNRES)\n## Proposed variants: P-LR-ATTNRES and S-LR-ATTNRES\n## Empirical results and contributions","[{\"question\":\"What problem do Attention Residuals (ATTNRES) address in Transformer residual connections?\",\"answer\":\"ATTNRES replaces fixed additive residual accumulation with learned attention across depth so the model can emphasize which previous sub-layer outputs to use when forming the next hidden state.\"},{\"question\":\"How does LR-ATTNRES reduce the cost or coupling introduced by ATTNRES?\",\"answer\":\"LR-ATTNRES keeps residual values full-dimensional but computes depth-routing keys in a lower r-dimensional space, decoupling routing from representation and avoiding score computation that scales with the full hidden width.\"},{\"question\":\"What are the differences between P-LR-ATTNRES and S-LR-ATTNRES?\",\"answer\":\"P-LR-ATTNRES learns a separate routing key via a projection, achieving the strongest validation performance among tested variants. S-LR-ATTNRES reuses the last r dimensions of each value as the routing key, removing the auxiliary projection path and lowering added FLOPs.\"}]",1784205545,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"low-rank-attention-residuals","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/low-rank-attention-residuals/85678/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem do Attention Residuals (ATTNRES) address in Transformer residual connections?","Question",{"text":75,"@type":76},"ATTNRES replaces fixed additive residual accumulation with learned attention across depth so the model can emphasize which previous sub-layer outputs to use when forming the next hidden state.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does LR-ATTNRES reduce the cost or coupling introduced by ATTNRES?",{"text":80,"@type":76},"LR-ATTNRES keeps residual values full-dimensional but computes depth-routing keys in a lower r-dimensional space, decoupling routing from representation and avoiding score computation that scales with the full hidden width.",{"name":82,"@type":73,"acceptedAnswer":83},"What are the differences between P-LR-ATTNRES and S-LR-ATTNRES?",{"text":84,"@type":76},"P-LR-ATTNRES learns a separate routing key via a projection, achieving the strongest validation performance among tested variants. S-LR-ATTNRES reuses the last r dimensions of each value as the routing key, removing the auxiliary projection path and lowering added FLOPs.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":28,"slug":118},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]