[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85791-en":3,"doc-seo-85791-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85791,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers","Large language models (LLMs) store factual knowledge in their parameters, and recent findings locate this knowledge in MLP layers. Existing models of fact-storing MLPs fail to explain why LLMs achieve information-theoretically optimal fact-storage rates. This work provides a theoretical account via decoding-margin analysis and introduces the first Transformer-compatible closed-form fact-storing MLP construction, achieving optimal scaling, supporting arbitrary I/O geometries, and enabling factual recall with fewer parameters. It also demonstrates modular fact editing by swapping MLPs.","arXiv :2607 . 10034v 1 [ cs .LG] 10 Jul 2026  \nMLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers  \nRoberto Garcia1 ∗ † Jerry Liu1 * † Ronny Junkins2 * Sabri Eyuboglu2 Atri Rudra3 Chris R´e2  \n1Institute for Computational & Mathematical Engineering, Stanford University  \n2Department of Computer Science, Stanford University  \n3Department of Computer Science and Engineering, University at Buffalo  \nAbstract  \nLarge language models (LLMs) store factual knowledge in their parameters.  \nWhile recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanistic interpretability models of fact-storage in LLMs fail to explain the surprising empirical phenomenon that they store facts at an information-theoretically optimal rate. In this work, we develop a theoretical account of this phenomenon. We develop the first Transformer-compatible fact-storing MLP closed-form construction that satisfies the following three properties empirically observed in LLMs: it (i) attains optimal fact storage scaling,(ii) handles arbitrary input/output geometries, and (iii) works inside Transformers. Key to our work is to analyze the decoding margin of MLPs, whereas prior work only studies MLP fact storage. Under isotropic embeddings, our construction achieves information-theoretically optimal storage capacity scaling and requires 10- 104 × fewer parameters at matched fact count than prior constructions. For arbitrary key and value embeddings, we show that our construction attains the same storage capacity scaling, up to penalization factors depending on the embedding geometries. Moreover, we demonstrate that our constructed MLPs can be used within Transformer blocks for factual recall tasks at optimal capacity scaling, requiring 15-63 × fewer parameters at matched fact count than prior constructions. Finally, as a proof-of-concept, we show that fact-storing MLPs enable modular fact editing by swapping a Transformer’s MLP with a new one.  \n1 Introduction  \nLarge language models (LLMs) achieve remarkable performance across domains such as mathematics, science, and law (Google DeepMind, 2024; Guha et al., 2023; Saab et al., 2024), in part because they can store vast amounts of knowledge in their parameters (Petroni et al., 2019; Meng et al., 2023a) . Prior work suggests that knowledge in Transformers is stored in Multi-Layer Perceptrons (MLPs) as key-value mappings, or facts (Geva et al., 2021; Dai et al., 2022) . However, despite these findings, fact-storing MLPs remain poorly understood.  \nWhile prior work has made important progress toward understanding and modeling fact storage in MLPs, existing models fail to capture three empirically observed properties of LLM fact storage: MLPs must (i) attain optimal fact storage scaling,(ii) handle arbitrary input/output geometries, and (iii) work inside Transformers. Mechanistic interpretability work (Geva et al., 2021; Dai et al., 2022) assumes MLPs store facts in individual neurons, but these models lead to suboptimal storage-capacity scaling. More recently, Nichaniet al. (2024) study LLM fact storage by introducing an MLP weight construction (NTK MLP)  \n∗ Equal first authors; order chosen by coin flip  \n†Corresponding authors: [robgarct@stanford.edu](robgarct@stanford.edu), [jerrywliu@stanford.edu](jerrywliu@stanford.edu).  \n‡Code is released at [https://github.com/HazyResearch/hebbian-mlps](https://github.com/HazyResearch/hebbian-mlps).  \nDecoding Margin MLP Transformer Block  \nγmin ↑  \nv1  \nMLP(k1  \nγmin ↓  \nlarge margin of error  \n)  \nsmall margin of error  \nMLP(k1) ≈ Emb(Paris)  \nFMLPW (x) = Xi=1 vi (ki , x)  \nk1 = Emb(capital of France)  \nStorage Capacity  \n|W| = F log(F)  \nMLP(k1 + ε) ≈ Emb(Paris)  \nMLPW (x) = Xi vi(ki , x)   \nStorage Capacity  \n|W| = F log(F)  \nk1 + ε = Emb(capital of France) + ε  \n= Attn(The capital of France is)  \nThe capital of France is  \nFigure 1: (A) Decoding margin illustration: Top plot shows an MLP with large decoding ma","cbCaipnDWbDWV6vP","https://ap.wps.com/l/cbCaipnDWbDWV6vP","pdf",1886413,1,82,"English","en",105,"# Abstract\n# Introduction\n# Decoding Margin and Transformer Block Storage Capacity\n# Hebbian Construction and Optimal Fact Storage Scaling\n# Transformer-Compatible Fact Recall and Modular Editing","[{\"question\":\"What problem does the paper address about LLM fact storage?\",\"answer\":\"It addresses why existing interpretability and constructive approaches cannot explain the observed empirical fact that LLMs store facts at information-theoretically optimal rates, even though fact storage is attributed to MLP layers.\"},{\"question\":\"What is the core technical idea introduced in this work?\",\"answer\":\"The paper analyzes the decoding margin of MLPs, instead of studying fact storage under noiseless key queries, enabling a closed-form MLP construction that Transformers can use for factual recall.\"},{\"question\":\"What guarantees and capabilities does the proposed MLP construction provide?\",\"answer\":\"Under isotropic embeddings it achieves information-theoretically optimal storage capacity scaling with far fewer parameters; for arbitrary key/value embeddings it matches the same scaling up to geometry-dependent penalties and can operate inside Transformer blocks, supporting modular fact editing by swapping MLPs.\"}]",1784206303,207,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"mlps-are-hebbians-constructing-efficient-fact-storing-mlps-for-transformers","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/mlps-are-hebbians-constructing-efficient-fact-storing-mlps-for-transformers/85791/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address about LLM fact storage?","Question",{"text":75,"@type":76},"It addresses why existing interpretability and constructive approaches cannot explain the observed empirical fact that LLMs store facts at information-theoretically optimal rates, even though fact storage is attributed to MLP layers.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is the core technical idea introduced in this work?",{"text":80,"@type":76},"The paper analyzes the decoding margin of MLPs, instead of studying fact storage under noiseless key queries, enabling a closed-form MLP construction that Transformers can use for factual recall.",{"name":82,"@type":73,"acceptedAnswer":83},"What guarantees and capabilities does the proposed MLP construction provide?",{"text":84,"@type":76},"Under isotropic embeddings it achieves information-theoretically optimal storage capacity scaling with far fewer parameters; for arbitrary key/value embeddings it matches the same scaling up to geometry-dependent penalties and can operate inside Transformer blocks, supporting modular fact editing by swapping MLPs.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]