[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-664774-105":59,"doc-detail-664774-en":129},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":122,"head_meta":124,"extra_data":126,"updated_unix":128},105,"en","decoder-hybrid-decoder-architecture-for-efficient-reasoning-with-long-generation","Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation","","Recent advances in language modeling show State Space Models (SSMs) can model sequences efficiently, and hybrid systems like Samba and YOCO outperform Transformers in some settings. However, prior work does not study efficiency benefits from sharing representations between SSM layers. This paper proposes GMU for gated memory sharing across layers and applies it in SambaY, a decoder-hybrid-decoder architecture. SambaY improves decoding efficiency, maintains linear pre-filling complexity, avoids explicit positional encoding, and yields lower irreducible loss than YOCO while scaling strongly to long contexts.",{"@graph":69,"@context":121},[70,84,104],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/decoder-hybrid-decoder-architecture-for-efficient-reasoning-with-long-generation/664774/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":98,"encodingFormat":97,"isAccessibleForFree":99,"interactionStatistic":100},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/decoder-hybrid-decoder-architecture-for-efficient-reasoning-with-long-generation/664774.png","ImageObject",300,407,{"name":92,"@type":93},"นรินทร์","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-29",true,{"@type":101,"interactionType":102,"userInteractionCount":8},"InteractionCounter",{"@type":103},"ViewAction",{"@type":105,"mainEntity":106},"FAQPage",[107,113,117],{"name":108,"@type":109,"acceptedAnswer":110},"What problem does the paper address for long-generation reasoning models?","Question",{"text":111,"@type":112},"The paper targets decoding inefficiency caused by the attention memory cost of cross-attentions during generation in long CoT reasoning scenarios.","Answer",{"name":114,"@type":109,"acceptedAnswer":115},"How does GMU enable representation sharing in SambaY?",{"text":116,"@type":112},"GMU provides a gated mechanism that shares memory readout states across layers, replacing part of the cross-attention computation with cheaper element-wise gating.",{"name":118,"@type":109,"acceptedAnswer":119},"What performance gains does SambaY report compared with a YOCO-based baseline?",{"text":120,"@type":112},"SambaY significantly improves decoding efficiency, preserves linear pre-filling time complexity, eliminates explicit positional encoding needs, and demonstrates lower irreducible loss with better scaling behavior under large compute regimes.","https://schema.org",{"og:url":83,"og:type":123,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":125,"canonical":83},"index,follow",{"doc_id":127,"site_id":62},664774,1793271982,{"code":4,"msg":5,"data":130},{"doc_id":127,"user_id":131,"nickname":92,"user_avatar":132,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":133,"file_id":134,"file_url":135,"file_type":136,"file_size":137,"view_count":8,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":138,"language":139,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":67,"update_tm":128,"read_time":143},2336475104957,"https://ap-avatar.wpscdn.com/avatar/22000c4c6bd8a5076e1?x-image-process=image/resize,m_fixed,w_180,h_180&k=1787554080175789136","Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation  \nLiliang Ren 1 Congcong Chen 1 Haoran Xu 1 Young Jin Kim 1 Adam Atkinson 1 Zheng Zhan 1 Jiankai Sun2 Baolin Peng 1 Liyuan Liu 1 Shuohang Wang 1 Hao Cheng 1  \nJianfeng Gao 1 Weizhu Chen 1 Yelong Shen 1  \n1Microsoft 2 Stanford University  \n{liliangren,[yeshe}@microsoft.com](yeshe}@microsoft.com)  \nAbstract  \nRecent advances in language modeling have demonstrated the effectiveness of State Space Models (SSMs) for efficient sequence modeling. While hybrid architectures such as Samba and the decoder-decoder architecture, YOCO, have shown promising performance gains over Transformers, prior works have not investigated the efficiency potential of representation sharing between SSM layers.  \nIn this paper, we introduce the Gated Memory Unit (GMU), a simple yet effective mechanism for efficient memory sharing across layers. We apply it to create SambaY, a decoder-hybrid-decoder architecture that incorporates GMUs in the cross-decoder to share memory readout states from a Samba-based self-decoder.  \nSambaY significantly enhances decoding efficiency, preserves linear pre-filling time complexity, and boosts long-context performance, all while eliminating the need for explicit positional encoding. Through extensive scaling experiments, we demonstrate that our model exhibits a significantly lower irreducible loss compared to a strong YOCO baseline, indicating superior performance scalability under large-scale compute regimes. Our largest model enhanced with Differential Attention, Phi4-mini-Flash-Reasoning, achieves significantly better performance than Phi4-mini-Reasoning on reasoning tasks such as Math500, AIME24/25, and GPQA Diamond without any reinforcement learning, while delivering up to 10× higher decoding throughput on 2K-length prompts with 32K generation length under the vLLM inference framework. We release our training codebase on open-source data at [https://github.com/microsoft/ArchScale](https://github.com/microsoft/ArchScale).  \n1 Introduction  \nState Space Models (SSMs) [GGR21, GGGR22, GD23, DG24], including linear attention [HDLL22, SDH+ 23, QHS+ 22, YWS+ 23, YWZ+ 24, YKH25] and modern Recurrent Neural Networks (RNNs)[BPS+ 24, BPL+ 25, PAA+ 23, GOA+ 24], have recently shown promising results for more efficient sequence modeling over Transformers [VSP+ 17] . While pure SSMs/RNNs offer computational advantages with their linear complexity, they exhibit a theoretical expressiveness gap on the incontext retrieval capability relative to Transformers [WDL24] . This gap can be bridged by hybrid architectures [LLB+ 24, DSF+ 24, RLL+ 25, WBR+ 24, DFD+ 25, Min25], even with the inclusion of as few as a single full attention layer [WDL24] . Recently, the decoder-decoder architecture, YOCO [SDZ+ 24], achieves linear complexity for long context processing through storing the Key-Value (KV) pairs from a single self-attention layer and re-using them across all subsequent layers through cross-attentions. In practice, YOCO has delivered substantial efficiency gains when processing user prompts with long sequences, but challenges remain; it does not mitigate the attention memory  \n39th Conference on Neural Information Processing Systems (NeurIPS 2025) .  \nI/O cost for its cross-attentions during the generation stage of the model responses. This limitation becomes particularly pronounced for modern large language models (LLMs) [Ope24, DA25] that generate extensively long Chains-of-Thought (CoTs) [WWS+ 22] for hard reasoning tasks.  \nIn this paper, we investigate the potential of representation sharing between SSM layers to enhance decoding efficiency. We introduce the Gated Memory Unit (GMU), a versatile, simple yet effective mechanism for efficient memory sharing across layers. Applying GMUs to the cross-decoder of YOCO, we create a novel model with our decoder-hybrid-decoder architecture named SambaY that uses Samba [RLL+ 25] for the self-decoder and replaces ","cbCaijkOYPdvvDUO","https://ap.wps.com/l/cbCaijkOYPdvvDUO","pdf",6135332,35,"English","# Abstract\n# Introduction\n## Related background on SSMs and hybrid models\n## YOCO and its cross-attention bottleneck\n## Proposed GMU and SambaY architecture\n## Scaling method (µP++) and experiments\n## Long-context retrieval evaluation\n## Differential Attention variant: Phi4-mini-Flash-Reasoning","[{\"question\":\"What problem does the paper address for long-generation reasoning models?\",\"answer\":\"The paper targets decoding inefficiency caused by the attention memory cost of cross-attentions during generation in long CoT reasoning scenarios.\"},{\"question\":\"How does GMU enable representation sharing in SambaY?\",\"answer\":\"GMU provides a gated mechanism that shares memory readout states across layers, replacing part of the cross-attention computation with cheaper element-wise gating.\"},{\"question\":\"What performance gains does SambaY report compared with a YOCO-based baseline?\",\"answer\":\"SambaY significantly improves decoding efficiency, preserves linear pre-filling time complexity, eliminates explicit positional encoding needs, and demonstrates lower irreducible loss with better scaling behavior under large compute regimes.\"}]","Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation | PDF",88]