[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84400-en":3,"doc-seo-84400-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},84400,7971461741311,"Ophelia","https://ap-avatar.wpscdn.com/avatar/74000253aff267980c6?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779345379180704826",8,"Research & Report","Super Weights in LLMs and the Failure of Selective Training","Recent work identified Super Weights—individual LLM parameters whose removal causes orders-of-magnitude performance degradation. This paper shows the effect does not generalize: isolating Super Weights and their local neighborhoods for training collapses accuracy to random-guessing on OLMo-1B and OLMo-7B. The failure is coordinate-specific rather than due to sparsity or module choice. Equal-size random-position training can improve, and rank-decomposed LoRA updates succeed. LoRA constrained around Super Weight coordinates yields statistically indistinguishable results, highlighting layer-wide coordination.","arXiv :2607 .08733v 1 [ cs .LG] 9 Jul 2026  \nSuper Weights in LLMs and the Failure of Selective Training  \nShreyas Subramanian, Adewale Akinfaderin, Akarsha Sehwag  \nAmazon Web Services  \n{subshrey, akinfaa, [akshseh](akshseh}@amazon.com)[}](akshseh}@amazon.com)[@amazon.com](akshseh}@amazon.com)  \nAbstract  \nRecent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation due to pruning Super Weights does not universally apply to all LLMs. Furthermore, if these parameters are so important, Super Weight-aware training should be effective. We show the opposite. Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B, and expanding to local neighborhoods of up to 36K parameters provides no improvement.  \nThe failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down proj layers instead improves over the baseline, so the collapse comes from targeting Super Weights, not from sparsity itself. Vanilla LoRA, updating every position inattention weight matrices through low-rank structure, succeeds with only 0.16% of parameters, and applying the same low-rank update to down proj succeeds as well. A 10-seed ablation confirms that constraining LoRA updates at positions corresponding to Super Weight coordinates yields statistically indistinguishable results. These findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning relies on structured decompositions over entire layers rather than targeting individually important weights.  \n1 Introduction  \nParameter-efficient fine-tuning (PEFT) methods like LoRA (Hu et al., 2022) achieve performance comparable to full fine-tuning while updating only 0.1–1% of parameters. Aghajanyan et al. (2021) showed that fine-tuning operates in a low intrinsic dimension, suggesting that even random low-dimensional subspaces suffice. A natural hypothesis follows: if random subspaces work, then deliberately targeting the most important parameters should work at least as well. Recent work by Yu et al. (2025) identified Super Weights, individual parameters whose removal increases perplexity by orders of magnitude, providing an ideal test case for this hypothesis. We show that the hypothesis is wrong. Training Super Weightsand their neighborhoods in isolation fails completely, and this failure is specific to those coordinates rather than to sparse training in general, establishing that parameter importance and parameter trainability in isolation are fundamentally different properties.  \nWe investigate these questions through experiments on OLMo-1B and OLMo-7B, spanning pruning, direct training, neighborhood training, LoRA-dproj-SW-freeze, LoRA-∆W-SWfreeze, and multi-seed ablation studies. We begin by addressing a methodological limitation of prior work: Yu et al. (2025) identified Super Weights from a single forward pass. We validate consistency across 20 diverse samples, finding that 9 weight positions create activation spikes in 100% of inputs, confirming Super Weights are structural properties of thepretrained model rather than input-specific artifacts.  \nOur experiments then reveal a clear pattern. Training only Super Weights (100–8,192 parameters) drops accuracy to random-guessing levels, and expanding to local neighborhoods of up to 36K parameters provides no improvement. Two controls on the down proj layers isolate the cause. Training an equal number of randomly chosen down proj positions (Super  \nWeight coordinates excluded) improves over the baseline rather than collapsing, so sparsity alone is not the problem. Applying a low-rank update to down proj also succeeds, so the module is not the problem either. The collapse is specific to targeting Super Weight coordinates and their entangled neighborhoods. Meanwhile, vanilla","cbCairbsQX6ffWe2","https://ap.wps.com/l/cbCairbsQX6ffWe2","pdf",434205,3,1,24,"English","en",105,"# Abstract\n# Introduction\n# Experiments and Methodology","[{\"question\":\"What are Super Weights in LLMs, and why are they important?\",\"answer\":\"Super Weights are individual parameters whose removal increases perplexity by orders of magnitude, making them an ideal test case for hypotheses about parameter importance.\"},{\"question\":\"Why does selective training on Super Weights fail?\",\"answer\":\"Training Super Weights (and even their local neighborhoods) in isolation collapses accuracy to random-guessing. Controls show the collapse is specific to targeting those coordinates, not to sparsity or the chosen module.\"},{\"question\":\"How does LoRA succeed where Super-Weight-only training fails?\",\"answer\":\"Vanilla LoRA updates weight-matrix positions via low-rank decomposition across entire layers, reaching significantly higher accuracy while training only a small fraction of parameters. Constraining LoRA updates at Super Weight coordinates has no measurable effect, indicating the problem lies in the isolation/coordination of those specific parameters.\"}]",1784195325,60,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"super-weights-in-llms-and-the-failure-of-selective-training","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/super-weights-in-llms-and-the-failure-of-selective-training/84400/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What are Super Weights in LLMs, and why are they important?","Question",{"text":75,"@type":76},"Super Weights are individual parameters whose removal increases perplexity by orders of magnitude, making them an ideal test case for hypotheses about parameter importance.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why does selective training on Super Weights fail?",{"text":80,"@type":76},"Training Super Weights (and even their local neighborhoods) in isolation collapses accuracy to random-guessing. Controls show the collapse is specific to targeting those coordinates, not to sparsity or the chosen module.",{"name":82,"@type":73,"acceptedAnswer":83},"How does LoRA succeed where Super-Weight-only training fails?",{"text":84,"@type":76},"Vanilla LoRA updates weight-matrix positions via low-rank decomposition across entire layers, reaching significantly higher accuracy while training only a small fraction of parameters. Constraining LoRA updates at Super Weight coordinates has no measurable effect, indicating the problem lies in the isolation/coordination of those specific parameters.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,109,114,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":29,"slug":108},5,"Comic","comic",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]