[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86372-en":3,"doc-seo-86372-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},86372,962075006959,"Anda","https://ap-avatar.wpscdn.com/avatar/e0002397efbe92a78e?_k=1776741047341049297",8,"Research & Report","Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models","Mixture-of-Experts (MoE) increases capacity using conditional computation, but Transformers lack a native primitive for knowledge lookup and therefore emulate retrieval inefficiently with computation. The work introduces conditional memory as a complementary sparsity axis, instantiated by Engram, which modernizes classic n-gram embedding for O(1) lookup. A Sparsity Allocation formulation reveals a U-shaped scaling law balancing MoE computation and static memory, enabling a 27B Engram model that outperforms iso-parameter and iso-FLOPs MoE baselines. Beyond retrieval gains, results show stronger improvements in reasoning and in code/math tasks, supported by mechanistic analysis and efficient deterministic addressing.","arXiv :2601 .07372v2 [ cs .CL] 12 Jul 2026  \nConditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models  \nXin Cheng1,2∗, Rui Tian2∗, Wangding Zeng2, Damai Dai2, Qinyu Chen2, Bingxuan Wang2, Zhenda Xie2, Kezhao Huang2, Xingkai Yu2 Chengqi Deng2, Shangyan Zhou2, Chenggang Zhao2, Zhewen Hao2 Yukun Li2, Han Zhang2, Zhengyan Zhang2, Yixuan Wei2, M.Y Xu2 Huishuai Zhang1, Dongyan Zhao1, Wenfeng Liang2  \n1 Peking University 2 DeepSeek-AI  \n{zhanghuishuai, [zhaody}@pku.edu.cn](zhaody}@pku.edu.cn)[ ](zhaody}@pku.edu.cn){chengxin, tianr22, zengwangding, [damai.dai}@deepseek.com](damai.dai}@deepseek.com)  \nAbstract  \nWhile Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic 􀀣-gram embedding for O (1) lookup. By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram) . Guided by this law, we scale Engram to 27B parameters, achieving superior performance over a strictly iso-parameter and iso-FLOPs MoE baseline. Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains (HumanEval +3.0; MATH +2.4) . Mechanistic analyses reveal that Engram relieves the backbone’s early layers from static reconstruction, effectively deepening the network for complex reasoning. Furthermore, by delegating local dependencies to lookups, it frees up attention capacity for global context, substantially boosting long-context retrieval (e.g., MultiQuery NIAH: 84.2 → 97.0) . Finally, Engram establishes infrastructure-aware efficiency: its deterministic addressing enables runtime prefetching from host memory, incurring negligible overhead. We envision conditional memory as an indispensable modeling primitive for nextgeneration sparse models. Code available at: [https://github. com/deepseek-ai/Engram](https://github. com/deepseek-ai/Engram).  \n1. Introduction  \nSparsity is a recurring design principle for intelligent systems, spanning from biological neural circuits (Lennie, 2003; Olshausen and Field, 1997) to modern Large Language Models (LLMs) . Currently, this principle is primarily realized through Mixture-of-Experts (MoE) (Dai et al., 2024; Shazeer et al., 2017), which scales capacity via conditional computation. Owing to its ability to drastically increase model size without proportional increases in compute, MoE has become the de facto standard for frontier models (Comanici et al., 2025; Guo et al., 2025; Kimi, 2025) .  \n*  \nEqual contribution.  \nDespite the success of this conditional computation paradigm, the intrinsic heterogeneity of linguistic signals suggests significant room for structural optimization. Specifically, language modeling entails two qualitatively different sub-tasks: compositional reasoning and knowledge retrieval. While the former demands deep, dynamic computation, a substantial portion of text—such as named entities and formulaic patterns—is local, static, and highly stereotyped (Constant et al., 2017; Erman, 2000) . The effectiveness of classical 􀀣-gram models (Brantset al., 2007; Liu et al., 2024b; Nguyen, 2024) in capturing such local dependencies implies that these regularities are naturally represented as computationally inexpensive lookups. Since standard Transformers (Vaswani et al., 2017) lack a native knowledge lookup primitive, current LLMs are forced to simulate retrieval through computation. For instance, resolving a common multi-token entity requires consuming multiple early layers of attention and feed-forward networks (Ghandeharioun et al., 2024; Ji","cbCaisEC8aB5IRr9","https://ap.wps.com/l/cbCaisEC8aB5IRr9","pdf",772599,6,1,35,"English","en",105,"# Abstract\n# Introduction\n## Conditional computation vs. static lookup\n## Conditional memory and Engram\n## Sparsity Allocation and scaling law","[{\"question\":\"What limitation of standard Transformers motivates the proposed approach?\",\"answer\":\"Standard Transformers do not provide a native knowledge lookup primitive, so LLMs must simulate retrieval using computation across early layers, leading to inefficient static reconstruction.\"},{\"question\":\"What is conditional memory and how is it instantiated in this work?\",\"answer\":\"Conditional memory is a sparsity axis that retrieves static embeddings via sparse lookup. It is instantiated by Engram, a module built on classic n-gram structure with modern adaptations for efficient indexing and integration.\"},{\"question\":\"What does the Sparsity Allocation problem and the resulting scaling law show?\",\"answer\":\"Given a fixed parameter budget, the Sparsity Allocation problem determines how to distribute capacity between MoE experts and Engram memory. Experiments reveal a U-shaped scaling law that optimizes the trade-off between neural computation and static memory.\"}]",1784211155,88,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"conditional-memory-via-scalable-lookup-a-new-axis-of-sparsity-for-large-language-models","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/conditional-memory-via-scalable-lookup-a-new-axis-of-sparsity-for-large-language-models/86372/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-27","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What limitation of standard Transformers motivates the proposed approach?","Question",{"text":76,"@type":77},"Standard Transformers do not provide a native knowledge lookup primitive, so LLMs must simulate retrieval using computation across early layers, leading to inefficient static reconstruction.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"What is conditional memory and how is it instantiated in this work?",{"text":81,"@type":77},"Conditional memory is a sparsity axis that retrieves static embeddings via sparse lookup. It is instantiated by Engram, a module built on classic n-gram structure with modern adaptations for efficient indexing and integration.",{"name":83,"@type":74,"acceptedAnswer":84},"What does the Sparsity Allocation problem and the resulting scaling law show?",{"text":85,"@type":77},"Given a fixed parameter budget, the Sparsity Allocation problem determines how to distribute capacity between MoE experts and Engram memory. Experiments reveal a U-shaped scaling law that optimizes the trade-off between neural computation and static memory.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]