[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82843-en":3,"doc-seo-82843-105":30,"detail-sidebar-cat-0-en-105":95},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82843,2336464648322,"Aria","https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488",8,"Research & Report","Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms","High-performance collective communication primitives are essential for HPC and ML workloads, yet common libraries like NCCL target only dense data and fail to reduce traffic for sparse messages. The work introduces sparsity-exploiting algorithms for all-gather, reduce-scatter, and all-reduce, supported by a new bitvector-based format Pici for low-overhead, fast (de)compression. Representations adapt during the collective to handle dynamic sparsity and fill-in. At 99% input sparsity, the approach delivers up to 5.25×, 2.5×, and 2.66× speedups over NCCL respectively.","Adaptive Space-efficient Collectives for Dynamic and Unstructured Sparsity on GPU Platforms  \nLannie Dalton Hough, Emir Gencer, Hoffmann Muki, Abhinav Bhatele  \nDepartment of Computer Science, University of Maryland, College Park, MD, USA {ldhough, egencer, [hoffmuki](hoffmuki}@umd.edu)[}](hoffmuki}@umd.edu)[@umd.edu](hoffmuki}@umd.edu), [bhatele@cs.umd.edu](bhatele@cs.umd.edu)  \narXiv :2607 .04676v 1 [ cs .DC] 6 Jul 2026  \nAbstract—High-performance collective communication primitives are necessary for a variety of high performance computing (HPC) and machine learning (ML) workloads. State-of-theart collective communication libraries such as NCCL optimize exclusively for dense data. However, when sending sparse data, we can reduce communication volume by not sending zeros. Unfortunately, explicitly handling sparsity introduces challenges such as format conversion overheads and densification during collectives that involve reductions. In this paper, we introduce sparsity-exploiting algorithms for three collectives that address these challenges: all-gather, reduce-scatter, and all-reduce. Our collective implementations are backed by a new bitvector-based format, Pici, designed for low overhead and fast (de)compression at moderate sparsities. Further, our algorithms adapt to the level of sparsity in data, modifying its representation during the course of the collective. At 99% input sparsity, our collectives achieve up to 5.25 ×, 2.5 ×, and 2.66 × speedups over NCCL for all-gather, reduce-scatter, and all-reduce, respectively.  \nI. MOTIVATION  \nSparsity, i.e., the prevalence of a large number of zeros or near-zero values in data, is common in certain parallel workloads, including high performance computing [1]–[4], but even more so in distributed deep learning [5]–[8] . Exploiting sparsity in data (both application input data and large matrices in memory) for improving the performance of computation and communication is common [9]–[11] . Communication optimizations center on sending only important information (e.g. nonzero values), often in collective operations such as all-gather, reduce-scatter, and all-reduce. As communication overheads in parallel applications can represent a significant fraction of execution time, particularly as applications scale to more GPUs and nodes, such optimizations can be particularly helpful for large messages and at high GPU counts.  \nHistorically, sparsity-aware kernels and communication collectives have targeted the highly-sparse regime (>99% zeros) or specific sparsity patterns. However, many workloads, such as training deep neural networks with pruning, often exhibit unstructured and moderate sparsity [5], [7] . In this paper, we introduce communication collectives that exploit unstructured sparsity across a wider range of sparsity levels and significantly improve communication performance.  \nThe task of implementing efficient sparsity-aware collectives (sparse collectives, in short) is challenging because of several features of sparse problems:  \nThis work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.  \n• Irregularity: Unlike many dense problems, sparse problems are highly variable in their characteristics. For example, both the degree of sparsity (percentage of data elements that are zeros) and the structure of sparsity (where the nonzeros occur) can vary and impact which algorithms, kernels, and sparse formats are effective.  \n• Dynamic changes: In the case of collectives that include a reduction such as reduce-scatter and all-reduce, the degree of sparsity can decrease during the collective (“dynamic sparsity”) . If processes contribute sparse data with differing nonzero indices, the density of intermediate and final outputs will increase (“fill-in”) . Thus, better support for moderate sparsity is required in such cases.  \n• Overheads: Conversion between dense and sparse formats (c","cbCaihYJ0tpyHJy1","https://ap.wps.com/l/cbCaihYJ0tpyHJy1","pdf",715963,3,1,12,"English","en",105,"# Motivation\n## Challenges in sparsity-aware collectives\n## Pici format and adaptive dynamic sparsity\n## Implementation and evaluation","[{\"question\":\"为什么传统 NCCL 对稀疏数据通信不够理想？\",\"answer\":\"NCCL 主要针对密集数据进行优化。发送稀疏数据时无法有效跳过零值，因此通信量难以显著降低。\"},{\"question\":\"文中提出的稀疏收集通信算法覆盖哪些集体操作？\",\"answer\":\"提出了面向 all-gather、reduce-scatter 和 all-reduce 的稀疏通信算法。\"},{\"question\":\"Pici 格式在算法中扮演什么作用？\",\"answer\":\"Pici 是一种基于 bitvector 的稀疏表示格式，设计目标是较低开销并支持在中等稀疏度下快速 (de)compression。\"},{\"question\":\"实验结果表明在什么稀疏度下能获得显著加速？\",\"answer\":\"在 99% 输入稀疏度下，三种 collectives 分别相对 NCCL 获得最高约 5.25×、2.5× 和 2.66× 的加速。\"}]",1784183365,30,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":90,"head_meta":92,"extra_data":94,"updated_unix":28},"adaptive-space-efficient-collectives-for-dynamic-and-unstructured-sparsity-on-gpu-platforms","",{"@graph":36,"@context":89},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/adaptive-space-efficient-collectives-for-dynamic-and-unstructured-sparsity-on-gpu-platforms/82843/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81,85],{"name":72,"@type":73,"acceptedAnswer":74},"为什么传统 NCCL 对稀疏数据通信不够理想？","Question",{"text":75,"@type":76},"NCCL 主要针对密集数据进行优化。发送稀疏数据时无法有效跳过零值，因此通信量难以显著降低。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"文中提出的稀疏收集通信算法覆盖哪些集体操作？",{"text":80,"@type":76},"提出了面向 all-gather、reduce-scatter 和 all-reduce 的稀疏通信算法。",{"name":82,"@type":73,"acceptedAnswer":83},"Pici 格式在算法中扮演什么作用？",{"text":84,"@type":76},"Pici 是一种基于 bitvector 的稀疏表示格式，设计目标是较低开销并支持在中等稀疏度下快速 (de)compression。",{"name":86,"@type":73,"acceptedAnswer":87},"实验结果表明在什么稀疏度下能获得显著加速？",{"text":88,"@type":76},"在 99% 输入稀疏度下，三种 collectives 分别相对 NCCL 获得最高约 5.25×、2.5× 和 2.66× 的加速。","https://schema.org",{"og:url":51,"og:type":91,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":93,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":96},[97,101,105,109,114,119,124,126,131,134,138],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":106,"show_sort_weight":107,"slug":108},"Exam",70,"exam",{"id":110,"doc_module":4,"doc_module_name":46,"category_name":111,"show_sort_weight":112,"slug":113},5,"Comic",60,"comic",{"id":115,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},6,"Technology",50,"technology",{"id":120,"doc_module":4,"doc_module_name":46,"category_name":121,"show_sort_weight":122,"slug":123},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":29,"slug":125},"research-report",{"id":127,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":129,"slug":130},9,"Religion & Spirituality",20,"religion-spirituality",{"id":129,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":129,"slug":133},"World Cup","world-cup",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":135,"slug":137},10,"Lifestyle","lifestyle",{"id":139,"doc_module":4,"doc_module_name":46,"category_name":140,"show_sort_weight":110,"slug":141},19,"General","general"]