[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82287-en":3,"doc-seo-82287-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82287,13056703019662,"Evangeline","https://ap-avatar.wpscdn.com/avatar/be000253a8e92610077?_k=1778726343310543188",8,"Research & Report","Super Tuning: From Activation Aware Pruning to Sparse Fine Tuning","Large language models stay expensive to fine-tune because full-parameter updates demand heavy memory, compute, and per-task storage. This work reuses saliency ideas from pruning to decide where a model should adapt. It proposes Super, a sparse PEFT method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score from calibration activations, and introduces Supra, a hybrid sparse+LoRA adapter with budget splitting. Experiments on Llama 3 variants show top accuracy among schedule-selected baselines and effective magnitude-only support.","arXiv :2607 .09287v 1 [ cs .LG] 10 Jul 2026  \nSuper-Tuning: From Activation-Aware Pruning to  \nSparse Fine-Tuning  \nIvan Ilin Philip Zmushko† Peter Richtárik  \nKAUST KAUST KAUST  \n[ivan.ilin@kaust.edu.sa](ivan.ilin@kaust.edu.sa) [zmushko.ph.a@gmail.com](zmushko.ph.a@gmail.com) [peter.richtarik@kaust.edu.sa](peter.richtarik@kaust.edu.sa)  \nAbstract  \nLarge language models (LLMs) remain expensive to fine-tune because fullparameter updates require substantial memory, compute, and per-task storage.  \nWe study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameterefficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3 .2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.  \n1 Introduction  \nLarge Language Models (LLMs) have become central to modern natural language processing, achieving strong results in a wide array of tasks including question answering, summarization, code generation, and reasoning. However, their impressive capabilities come with a substantial cost: full fine-tuning of such models requires considerable computational resources, high memory consumption, and significant storage for each downstream task. This inefficiency becomes particularly problematic when deploying models in real-world settings with resource constraints or personalization requirements.  \nTo address these limitations, parameter-efficient fine-tuning (PEFT) methods have been proposed. These approaches update only a small fraction of the model’s parameters, leaving the majority of weights frozen. Among the most successful PEFT techniques is LoRA [Hu et al., 2021], which injects low-rank trainable matrices into existing layers. LoRA enables tuning with minimal parameter overhead and has been widely adopted for its balance of efficiency and performance.  \nBeyond low-rank adaptation, recent work has explored the use of sparse adaptation, where only a small, selected subset of existing weights are updated. Methods such as SIFT [Song et al., 2023] and RoSA [Nikdan et al., 2024] select salient weights for fine-tuning based on importance scores or gradient signals. PaFi [Liao et al., 2023] instead shows that a task-agnostic sparse mask can be generated without data or gradients by using only pretrained weight magnitudes. NeFT [Xu et al.,  \n†Work done while Philip Zmushko was an intern at KAUST; he is now affiliated with ISTA and was affiliated with Yandex Research before coming to KAUST.  \n1Code repository: [https://github.com/vectozavr/SuperTuning](https://github.com/vectozavr/SuperTuning).  \nPreprint.  \n2024] takes this further by identifying and training only the most critical neurons. SpIEL [Ansellet al., 2024] introduces scalable sparse tuning with structured expert layers, and SAT [Ma et al., 2024] proposes sparsity-accelerated training with carefully selected updates. S2 FT [Yang et al., 2024] combines sparsity with structured decomposition for efficient and generalizable tuning, while GIFTSW [Zhelnin et al., 2024] uses noise-injected fine-tuning on salient weights to improve robustness. Recent salience-aware sparse PEFT work further studies static and dynamic","cbCaie1vpqh2IUTP","https://ap.wps.com/l/cbCaie1vpqh2IUTP","pdf",676550,1,26,"English","en",105,"# Abstract\n# Introduction\n## Parameter-efficient fine-tuning and LoRA\n## Sparse adaptation methods and saliency metrics\n## Motivation and contributions of Super and Supra","[{\"question\":\"Why is full fine-tuning of large language models expensive?\",\"answer\":\"Full-parameter updates require substantial memory and compute and also add significant per-task storage. This becomes problematic under real-world resource constraints or personalization needs.\"},{\"question\":\"How does Super choose the sparse support for fine-tuning?\",\"answer\":\"Super fixes a small trainable support by computing a Wanda-style activation-weighted magnitude score from a calibration pass. It then selects weights under TopK or BottomK regimes based on these scores.\"},{\"question\":\"What is the difference between Supra and Super?\",\"answer\":\"Supra combines the sparse update from Super with LoRA in a hybrid adapter. It preserves a matched trainable-parameter budget using a simple budget-splitting rule.\"}]",1784179404,66,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"super-tuning-from-activation-aware-pruning-to-sparse-fine-tuning","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/super-tuning-from-activation-aware-pruning-to-sparse-fine-tuning/82287/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"Why is full fine-tuning of large language models expensive?","Question",{"text":74,"@type":75},"Full-parameter updates require substantial memory and compute and also add significant per-task storage. This becomes problematic under real-world resource constraints or personalization needs.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does Super choose the sparse support for fine-tuning?",{"text":79,"@type":75},"Super fixes a small trainable support by computing a Wanda-style activation-weighted magnitude score from a calibration pass. It then selects weights under TopK or BottomK regimes based on these scores.",{"name":81,"@type":72,"acceptedAnswer":82},"What is the difference between Supra and Super?",{"text":83,"@type":75},"Supra combines the sparse update from Super with LoRA in a hybrid adapter. It preserves a matched trainable-parameter budget using a simple budget-splitting rule.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]