[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83264-en":3,"doc-seo-83264-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83264,13056703019662,"Evangeline","https://ap-avatar.wpscdn.com/avatar/be000253a8e92610077?_k=1778726343310543188",8,"Research & Report","PALS Percentile-Aware Layerwise Sparsity for LLM Pruning","One-shot pruning methods such as Wanda and SparseGPT apply a uniform sparsity ratio across all transformer layers, disregarding that layer importance varies. PALS (Percentile-Aware Layerwise Sparsity) adapts per-layer sparsity using the 99th percentile of activation magnitudes, constrained within a ±5% band around the target. On LLaMA-2-7B at 50% sparsity, PALS reduces WikiText-2 perplexity from 12.92 to 10.96 (p \u003C 0.001, nine runs). Performance depends on architecture, and gradient-based allocation underperforms discrete activation-based removal.","PALS: Percentile-Aware Layerwise Sparsity for LLM  \nPruning  \nYazdan Jamshidi  \nPalo Alto Networks [yjamshidi@paloaltonetworks.com](yjamshidi@paloaltonetworks.com)  \nAlexey Shvets  \nPalo Alto Networks [ashvets@paloaltonetworks.com](ashvets@paloaltonetworks.com)  \narXiv :2607 .07557v 1 [ cs .CL] 8 Jul 2026  \nAbstract  \nOne-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to ±5% around the target ratio. On LLaMA-2-7B at 50% sparsity, PALS achieves  \n10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs, p \u003C 0.001) . The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation—the seemingly more principled approach—produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no finetuning.  \n1 Introduction  \nLarge language models routinely reach billions of parameters. A 7B-parameter model takes roughly 13 GB in half precision and processes billions of multiply-accumulate operations per token, which puts real-time inference out of reach on modest hardware. Post-training pruning offers a direct way to cut both memory and compute: remove a fraction of the weights and run inference on the sparser model, without retraining from scratch.  \nRecent one-shot pruning methods have made this practical for LLMs. Wanda Sun et al. [2024] scores each weight by the product of its magnitude and the mean absolute activation flowing through it (|wij | · mean(|ai |)), then drops the lowest-scoring fraction. SparseGPT Frantar and Alistarh [2023] solves a heavier problem—layer-wise sparse regression with approximate Hessian information—but in practice lands at similar perplexity on most benchmarks. Both methods have become standard baselines.  \nOne thing both share is that they enforce uniform sparsity: if the target is 50%, every one of the 32 layers in LLaMA-2 is pruned to exactly 50% . This sits poorly with what we know about transformer layers. Attention heads vary widely in importance Voita et al. [2019], Michel et al. [2019] . Early layers tend to build generic representations; middle layers are often more redundant; late layers specialize for the output vocabulary. Pruning all of them equally seems like it should leave performance on the table—and it does.  \nWe propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts the sparsity ratio for each layer based on activation statistics gathered from a small calibration set. The idea is direct: compute the 99th percentile of activation magnitudes for each layer, normalize these scores, and use them to shift each layer’s sparsity target up or down. Layers with large activation outliers—a sign of important information flow—get pruned less; layers with uniformly small activations get pruned more. A ±5% band around the target prevents any single layer from being starved or overstuffed.  \nPreprint.  \nThe procedure slots into Wanda’s pipeline with almost no extra cost, since Wanda already collects the activations.  \nBefore settling on activations, we tried the seemingly more natural approach of using gradient magnitudes to estimate layer importance. This failed decisively. Gradient-based allocation produced 47 perplexity on LLaMA-2-7B, compared to 24 for random allocation and 11 for our activation-based scheme. We believe gradients reflect optimization dynamics—how much the loss would change under an infinitesimal perturbation—rather than the actual impact of removing a large fraction of weights. But the scale of the failure was not something we anticipated.  \nOn LLaMA-2-7B at 50% sparsity, PALS brings WikiText-2 per","cbCaieb0B8BCnxwb","https://ap.wps.com/l/cbCaieb0B8BCnxwb","pdf",312493,3,1,12,"English","en",105,"# Abstract\n# Introduction\n# Related Work\n## One-Shot Pruning for LLMs\n## Structured Pruning","[{\"question\":\"What problem does PALS address in one-shot LLM pruning?\",\"answer\":\"PALS addresses the limitation of uniform sparsity allocation across transformer layers, which ignores known differences in layer importance.\"},{\"question\":\"How does PALS determine the sparsity ratio for each layer?\",\"answer\":\"PALS computes the 99th percentile of activation magnitudes per layer on a small calibration set, normalizes these scores, and shifts each layer’s sparsity target up or down with a ±5% bound around the target ratio.\"},{\"question\":\"How effective is PALS compared with Wanda and gradient-based allocation?\",\"answer\":\"On LLaMA-2-7B at 50% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda; gradient-based allocation performs worse than random allocation and also worse than the activation-based scheme.\"}]",1784186371,30,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"pals-percentile-aware-layerwise-sparsity-for-llm-pruning","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/pals-percentile-aware-layerwise-sparsity-for-llm-pruning/83264/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-21","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does PALS address in one-shot LLM pruning?","Question",{"text":75,"@type":76},"PALS addresses the limitation of uniform sparsity allocation across transformer layers, which ignores known differences in layer importance.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does PALS determine the sparsity ratio for each layer?",{"text":80,"@type":76},"PALS computes the 99th percentile of activation magnitudes per layer on a small calibration set, normalizes these scores, and shifts each layer’s sparsity target up or down with a ±5% bound around the target ratio.",{"name":82,"@type":73,"acceptedAnswer":83},"How effective is PALS compared with Wanda and gradient-based allocation?",{"text":84,"@type":76},"On LLaMA-2-7B at 50% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda; gradient-based allocation performs worse than random allocation and also worse than the activation-based scheme.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":29,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]