[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82047-en":3,"doc-seo-82047-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82047,7971461740909,"Levi","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices","With large language models (LLMs) widely deployed, inference cost has become a primary bottleneck. Pruning can add sparsity to weight matrices, but preserving model quality usually restricts pruning to moderate unstructured sparsity (around 50%), where existing sparse matrix multiplication (SpMM) GPU kernels cannot beat dense libraries. This work introduces a GPU inference approach with a three-layer matrix storage format and a joint sparse-tensor-core plus CUDA-core SpMM kernel to improve execution and overlap compute with memory. Results show first-of-its-kind speedups over prior dense and sparse systems.","Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices  \nTao Lu§,*, Haoyu Wang¶,*, Zonghui Wang¶,†, Keshen Xiang¶ , Jiaheng Zhang§,†, Wenzhi Chen¶  \n§ National University of Singapore,¶ Zhejiang University  \nSingapore, China  \n{lutao,[jhzhang}@nus.edu.sg](jhzhang}@nus.edu.sg),{whaoyu,zhwang,keshen, [chenwz}@zju.edu.cn](chenwz}@zju.edu.cn)  \narXiv :2607 .08786v1 [ cs .LG] 13 Jun 2026  \nAbstract  \nWith the growing deployment of large language models (LLMs), LLM inference cost has become a key challenge. Pruning techniques that introduce sparsity into weight matrices can accelerate inference. However, maintaining model quality typically limits pruning to moderate unstructured sparsity (around 50%). At these sparsity levels, none of the existing GPU kernels for sparse matrix multiplication (SpMM) can outperform their dense counterparts. This paper proposes an efficient GPU inference method for LLMs with moderate sparsity. We propose a three-layer matrix storage format comprising: (i) a Sparse-TC layer enabling sparse tensor cores to accelerate SpMM; (ii) a Slot-Filling layer using parallel differential distance for matrix compression while supporting low-cost on-chip decoding; (iii) a lightweight Residual Layer ensuring correct SpMM computation. Building on this format, we design a SpMM kernel that jointly utilizes sparse tensor cores and CUDA cores. This design enables an efficient execution pipeline and overlaps on-chip computation with memory access. Evaluations show that our work is the first to outperform dense matrix multiplication on modern GPUs equipped with high-bandwidth memory (HBM) . It achieves up to 1.64 × kernel-level speedup over SpInfer (EuroSys’25, Best paper) and up to 1.41 × end-to-end speedups over FlashLLM (VLDB’24) . Our source code: [https://github.com/moui0/cudac](https://github.com/moui0/cudac).  \nACM Reference Format:  \nTao Lu, Haoyu Wang, Zonghui Wang, Keshen Xiang, Jiaheng Zhang, Wenzhi Chen. 2026. Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices. In 63rd ACM/IEEE Design Automation Conference (DAC’26), July 26–29, 2026, Long Beach, CA, USA. ACM, New York, NY, USA, 7 pages. [https://doi.org/10.1145/3770743.3804227](https://doi.org/10.1145/3770743.3804227)  \n1 Introduction  \nWith the growing adoption of large language models (LLMs) [10, 25, 28, 34] in applications such as natural language understanding, text generation, and code completion, the high computational cost of LLM inference has become a critical concern. To mitigate this, recent works, such as SparseGPT [8], Wanda [27], and RIA [35], have proposed effective pruning strategies that remove less important values from the weight matrices. These approaches aim to accelerate inference while preserving model quality.  \n* Equal contributions.†Corresponding authors.  \nThis work is licensed under a Creative Commons Attribution 4 .0 International License. DAC’26, Long Beach, CA, USA  \n© 2026 Copyright held by the owner/author(s) .  \nACM ISBN 979-8-4007-2254-7/2026/07  \n[https://doi.org/10.1145/3770743.3804227](https://doi.org/10.1145/3770743.3804227)  \nTable 1: Perplexity scores of prominent LLM pruning schemes, Wanda [27] and RIA [35], evaluated across various sparsity levels and topologies on the WikiText-2 dataset, where a lower perplexity score indicates better performance.  \n\n| Sparsity | Topology | Scheme | Llama\u003Cbr>7B | Llama\u003Cbr>13B | Opt\u003Cbr>1.3B |\n| --- | --- | --- | --- | --- | --- |\n| Dense | - | - | 5.47 | 4.88 | 14.6 |\n| 50% | unstructured | Wanda | 7.79 | 6.28 | 18.5 |\n|  |  | RIA | 6.88 | 5.95 | 18.9 |\n|  | 2:4 | Wanda | 11.6 | 9.69 | 28.3 |\n|  |  | RIA | 11.3 | 8.44 | 27.4 |\n| 60% | unstructured | Wanda | 15.3 | 9.63 | 38.8 |\n|  |  | RIA | 10.4 | 7.84 | 26.2 |\n| 70% | unstructured | Wanda | 214.9 | 105.0 | 231.2 |\n|  |  | RIA | 68.8 | 52.0 | 98.5 |\n\nHowever, unlike small-scale models, where high sparsity can be introduced with minimal im","cbCaieVUJPHIH1VT","https://ap.wps.com/l/cbCaieVUJPHIH1VT","pdf",789181,2,1,7,"English","en",105,"# Introduction\n## Motivation: Pruning and inference cost\n## Limitations of existing SpMM kernels\n## Challenges of moderate unstructured sparsity","[{\"question\":\"Why do moderate unstructured sparsity levels matter for LLM pruning?\",\"answer\":\"The document states that pruning that maintains model quality typically limits unstructured sparsity to around 50%. At these levels, prior sparse SpMM GPU kernels fail to outperform dense counterparts.\"},{\"question\":\"What are the main challenges of accelerating SpMM under moderate unstructured sparsity?\",\"answer\":\"It highlights tensor core incompatibility with irregular unstructured sparsity, large metadata overhead in formats like CSR at moderate sparsity, and significant on-chip decoding overheads during matrix compression.\"},{\"question\":\"How does the proposed method accelerate GPU inference for LLMs with moderate sparsity?\",\"answer\":\"It uses a three-layer matrix storage format (Sparse-TC, Slot-Filling, and a Residual Layer) and designs a SpMM kernel that jointly leverages sparse tensor cores and CUDA cores, enabling an efficient execution pipeline with overlap between on-chip computation and memory access.\"}]",1784177799,18,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"accelerating-gpu-inference-of-large-language-models-with-moderately-unstructured-sparse-weight-matrices","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/accelerating-gpu-inference-of-large-language-models-with-moderately-unstructured-sparse-weight-matrices/82047/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-20","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do moderate unstructured sparsity levels matter for LLM pruning?","Question",{"text":75,"@type":76},"The document states that pruning that maintains model quality typically limits unstructured sparsity to around 50%. At these levels, prior sparse SpMM GPU kernels fail to outperform dense counterparts.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What are the main challenges of accelerating SpMM under moderate unstructured sparsity?",{"text":80,"@type":76},"It highlights tensor core incompatibility with irregular unstructured sparsity, large metadata overhead in formats like CSR at moderate sparsity, and significant on-chip decoding overheads during matrix compression.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the proposed method accelerate GPU inference for LLMs with moderate sparsity?",{"text":84,"@type":76},"It uses a three-layer matrix storage format (Sparse-TC, Slot-Filling, and a Residual Layer) and designs a SpMM kernel that jointly leverages sparse tensor cores and CUDA cores, enabling an efficient execution pipeline with overlap between on-chip computation and memory access.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]