[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83231-en":3,"doc-seo-83231-105":30,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83231,962075114765,"Quinn","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity","Sparse Delta Memory (SDM) presents an architecture that scales the hidden state of gated linear RNNs to much higher capacity while keeping compute and memory per token constant. Linear attention models use fixed state size but underperform softmax transformers on long-context recall because increasing state capacity raises FLOPs. SDM replaces the dense key-value outer product with sparse reads and writes to a large explicit memory, improving in-context learning and long-context retrieval under isoFLOP and parameter-matched settings, and further benefits from a learned initial memory state.","arXiv :2607 .07386v 1 [ cs .LG] 8 Jul 2026  \nSparse Delta Memory: Scaling the State of Linear RNNs through Sparsity  \nLoïc Cabannes 1 ,2 , Pierre-Emmanuel Mazaré 1 , Gergely Szilvasy 1 , Matthijs Douze 1 , Maria Lomeli 1 , Ilze Amanda Auzina 1 ,3 , Justin Carpentier2 , Gabriel Synnaeve 1 , Hervé Jégou 1  \n1Meta FAIR, 2Inria Paris & ENS-PSL University, 3University of Tübingen  \nLinear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory(SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.  \nDate: July 9, 2026  \nCode: [https://github.com/facebookresearch/sparse-delta-memory](https://github.com/facebookresearch/sparse-delta-memory)  \n1 Introduction  \nAs frontier models continue to progress, they are leveraged in increasingly more complex tasks. In particular, the emergence of agentic settings involve sustained reasoning over long contexts, including software engineering, research assistance, and personal assistants.  \nThese applications demand memory mechanisms that preserve long-range dependencies across extended interactions. In the standard transformer architectures equipped with a vanilla softmax attention, the interaction memory is stored in a Key-Value (KV) cache. It effective on long-context tasks, yet the KV cache and therefore the compute and memory per token all grow linearly with the sequence length,(Fig. 1 green line) . This unbounded growth limits in-context learning over very long sequences, such as entire codebases or extended reasoning traces, which are increasingly central to autonomous agents, but also videos which play a central role notably in world models and robotics.  \n1GB  \n100MB  \n10MB  \n1MB  \n100KB  \n108 109  \nState size  \nFLOPs / token  \nFigure 1 State size vs FLOPs per token  \n(global layer, 1.4B) . KV cache scales linearly with the sequence length. Our SDM approach offers a large state with constant FLOPs.  \nA possible way to avoid this growing KV cache is to replace the explicit storage of all past tokens with a compressed recurrent state. Recurrent Neural Networks (RNNs), including State Space Models (Gu & Dao, 2024) and Linear Attention variants (Katharopoulos et al., 2020; Beck et al., 2024), rely on this strategy: they compress information into a fixed-size hidden state, maintaining constant memory and compute per token  \nregardless of sequence length. This enables the processing of arbitrarily long contexts without an explicit token limit. However, their extremely small state sizes limit recall capability compared to transformers (Fu et al., 2023) . Indeed, Arora et al. (2025) show that long-context performance is fundamentally bounded by the hidden state size. Simply increasing the RNN memory size would improve recall, but modern linear RNNs like Mamba2 (Dao & Gu, 2024) and Gated DeltaNet (GDN) (Yang et al., 2025) are bottlenecked by dense state updates that become prohibitively expensive as the state size grows.  \nTo address the limitations of dense state updates, we introduce Sparse Delta Memory (SDM), a novel ar","cbCaigeKmsUe5o6A","https://ap.wps.com/l/cbCaigeKmsUe5o6A","pdf",905063,3,1,20,"English","en",105,"# Introduction\n# Background","[{\"question\":\"What does learning the initial SDM memory enable?\",\"answer\":\"By learning the initial state of the SDM memory and treating it as a parametric memory, the model stores meaningful pretraining knowledge. This yields further improvements across common-knowledge and reasoning tasks and lowers training loss compared with Gated DeltaNet under isoFLOP comparisons.\"}]",1784186102,50,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":28},"sparse-delta-memory-scaling-the-state-of-linear-rnns-through-sparsity","",{"@graph":36,"@context":77},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/sparse-delta-memory-scaling-the-state-of-linear-rnns-through-sparsity/83231/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"What does learning the initial SDM memory enable?","Question",{"text":75,"@type":76},"By learning the initial state of the SDM memory and treating it as a parametric memory, the model stores meaningful pretraining knowledge. This yields further improvements across common-knowledge and reasoning tasks and lowers training loss compared with Gated DeltaNet under isoFLOP comparisons.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":84},[85,89,93,97,102,106,111,114,118,121,125],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":46,"category_name":104,"show_sort_weight":29,"slug":105},6,"Technology","technology",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":112,"slug":113},30,"research-report",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":22,"slug":117},9,"Religion & Spirituality","religion-spirituality",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":119,"show_sort_weight":22,"slug":120},"World Cup","world-cup",{"id":122,"doc_module":4,"doc_module_name":46,"category_name":123,"show_sort_weight":122,"slug":124},10,"Lifestyle","lifestyle",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":127,"show_sort_weight":98,"slug":128},19,"General","general"]