[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82361-en":3,"doc-seo-82361-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82361,687197207919,"Theodora","https://ap-avatar.wpscdn.com/avatar/a000253d6f5f7c60be?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779446848396160552",8,"Research & Report","Neural Collapse Is Forbidden: Information Floors in Language Models","Neural collapse in language models is commonly interpreted as incomplete representation collapse, but within-class variance is instead framed as information storage governed by a precise law. Across 14 models and diverse token-type regimes, within-token context variability accounts for 79–91% of total variance while category-level structure remains only 4–12%. Token-level weight decay yields an imbalanced K-class learning objective ordered by type count, not occurrence mass. A converse information floor, proven for binary categories, forces dispersion within categories to scale with conditional mutual information I(token; context | category).","arXiv :2607 .09487v 1 [ cs .LG] 10 Jul 2026  \nNeural Collapse Is Forbidden: Information Floors in Language Models  \nBruno Abrahao  \nNYU Shanghai  \nLeonard N. Stern School of Business, New York University  \n[bd58@nyu. edu](bd58@nyu. edu)  \nJuly 2026  \nAbstract  \nWithin-class variance in language-model representations is commonly read as incomplete neural collapse. We argue it is allocated information storage, and that the allocation obeys a law. A one-line centering identity voids a family of simplex equiangular-tight-frame claims, including our own earlier ones; in dimensionless variance shares across 14 models, macro-category structure carries only 4–12% of representational variance and within-token context carries 79–91%, stable across a 100x parameter range. On the theory side, token-level weight decay penalizes a category in proportion to its type count, not its occurrence mass, reducing next-token prediction to animbalanced K-class problem whose optimum orders category norms by type count. A converse floor, proved for binary categories, forces within-category dispersion to be at least proportional to the conditional mutual information I (token; context | category) . The law holds: identity dispersion, not total variance, tracks this information across every tested model and partition, under a model-free estimate and even across models, where one model’s information predicts another’s dispersion; and over pretraining the category share overshoots, decays, and partially recovers, because the information it must carry never left.  \n1 Introduction  \nIn image classification, the terminal phase of training is when geometry tightens: within-class variability collapses and class means settle into a simplex equiangular tight frame, the neural collapse (NC) of Papyan et al. [12] . Language models invert this script. Measuring the geometry of hidden states over public pretraining checkpoints, we find that category-level structure crystallizes almost immediately, overshoots, then partially dissolves over the long remainder of training, and finally, in most model sizes, partially returns, all while perplexity improves monotonically. The terminal phase of a language model does not finish collapsing its representations; it reallocates them. Figure 1 previews the three findings this inversion opens.  \nWe argue that the resolution of this apparent paradox is informational rather than geometric, and that it reframes what within-class variance in language models is. The prevailing reading, inherited from classification, treats residual within-class variability as unfinished collapse: noise that more scale or more training should remove. On measurement grounds alone this reading fails. Using a dimensionless three-level decomposition of representational variance (within-token context variability, token identity within category, and category structure), we find across 14 models from three families (GPT-2, Pythia, Qwen2.5; 70M–6.9B parameters) that within-token context variability carries 79–91% of total variance (69% in one degenerate model), token identity 4–13%,  \ncontext  \nidentity  \ncategory  \n(a) variance is mostly stored context  \n\n|  |  | 84% |\n| --- | --- | --- |\n| \u003Cbr> 8%\u003Cbr> 8% |  |  |\n\n0.0 0.5 1.0 share of total variance  \nidentity dispersion Bk  \n103  \n102  \n101  \n(b) dispersion tracks information (r = 0.755)  \n0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5  \n̂  \nconditional information I(c; ctx ∣ Sk)  \ncategory share  \n0.200 0.175 0.150 0.125 0.100 0.075 0.050 0.025  \n0.000  \n(c) rise, fall, partial recovery  \n106 107 108 109 1010 1011 1012 training tokens (Pythia dashed, OLMo-2 solid)  \nFigure 1: (a) Across 13 models, variance is dominated by within-token context; macro-category structure is a thin slice, stable across a 100x parameter range. (b) Within-category identity dispersion tracks model-realized conditional information ˆI(c; ctx | Sk) (pooled partial r = 0 . 755), the dispersion channel the information floor of Section 3 makes necessary","cbCaihU7TrESVp84","https://ap.wps.com/l/cbCaihU7TrESVp84","pdf",548263,1,26,"English","en",105,"# Abstract\n# Introduction\n# Variance Decomposition and Empirical Findings\n## Proposed Informational Explanation\n## Variance Allocation Across Models","[{\"question\":\"How does the paper reinterpret within-class variance in language model representations?\",\"answer\":\"It argues within-class variance is allocated information storage rather than unfinished neural collapse. The allocation follows a specific rule tied to how information is tracked across representations.\"},{\"question\":\"What fraction of representational variance comes from within-token context versus category structure?\",\"answer\":\"Across 14 tested models, within-token context variability carries about 79–91% of total variance, while macro-category structure carries only about 4–12%.\"},{\"question\":\"What theoretical mechanism produces the “information floor” and dispersion behavior?\",\"answer\":\"Token-level weight decay penalizes categories proportionally to type count rather than occurrence mass, reducing next-token prediction to an imbalanced K-class problem. A converse floor for binary categories then forces within-category dispersion to scale with conditional mutual information I(token; context | category).\"}]",1784179919,66,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"neural-collapse-is-forbidden-information-floors-in-language-models","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/neural-collapse-is-forbidden-information-floors-in-language-models/82361/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"How does the paper reinterpret within-class variance in language model representations?","Question",{"text":75,"@type":76},"It argues within-class variance is allocated information storage rather than unfinished neural collapse. The allocation follows a specific rule tied to how information is tracked across representations.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What fraction of representational variance comes from within-token context versus category structure?",{"text":80,"@type":76},"Across 14 tested models, within-token context variability carries about 79–91% of total variance, while macro-category structure carries only about 4–12%.",{"name":82,"@type":73,"acceptedAnswer":83},"What theoretical mechanism produces the “information floor” and dispersion behavior?",{"text":84,"@type":76},"Token-level weight decay penalizes categories proportionally to type count rather than occurrence mass, reducing next-token prediction to an imbalanced K-class problem. A converse floor for binary categories then forces within-category dispersion to scale with conditional mutual information I(token; context | category).","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]