[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83663-en":3,"doc-seo-83663-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},83663,3848291630094,"Emma Wilson","https://eur-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45",8,"Research & Report","Tile-Level Activation Overlap for Efficient LLM Inference","SwiGLU dominates the MLP activation in modern large language models, yet materializing its intermediate tensors consumes 9–37% of MLP execution time. This work introduces two complementary CUTLASS-based SM90 kernels that fuse SwiGLU into GeMM at tile level, overlapping Swish computation with Up-tile loading and interleaving gated activation with tile stores. On Qwen-2.5 (0.5B–72B) running on NVIDIA H100, the kernels reach up to 2.47× speedup over PyTorch, shift workloads from memory- to compute-bound, and obtain 79.5% peak BF16 utilization, with zero numerical mismatches versus cuBLAS error rates of 4.5–11%.","Tile-Level Activation Overlap for Efficient LLM  \nInference  \nAbhinav Jangda  \nMicrosoft Research  \n[ajangda@microsoft.com](ajangda@microsoft.com)  \nTyler Sorensen  \nMicrosoft Research  \n[tsorensen@microsoft.com](tsorensen@microsoft.com)  \nSebastian Burckhardt  \nMicrosoft Research  \n[sburckha@microsoft.com](sburckha@microsoft.com)  \narXiv :2607 .02521v1 [ cs .DC] 5 May 2026  \nJianlan YE  \nMicrosoft  \n[jianlanye@microsoft.com](jianlanye@microsoft.com)  \nChaoyin Li  \nMicrosoft  \n[chaoyl@microsoft.com](chaoyl@microsoft.com)  \nAtul Gupta  \nMicrosoft  \n[atul.gupta@microsoft.com](atul.gupta@microsoft.com)  \nAbstract  \nSwiGLU is the dominant MLP activation in modern large language models, yet its intermediate tensor materialization costs 9–37% of MLP execution time. We present two complementary CUTLASS-based SM90 kernels that fuse SwiGLU into GeMM at the tile level. Kernel-1 overlaps Swish computation on the Gate accumulator with Up-tile loading using the Pingpong warp-specialized schedule;  \nKernel-2 interleaves SwiGLU with tile stores via a custom Epilogue Visitor Tree.  \nEvaluated on Qwen-2.5 models (0.5B–72B) on NVIDIA H100, our kernels achieve up to 2.47 × speedup over PyTorch, shifting workloads from memory-bound to compute-bound and reaching 79.5% peak BF16 utilization. We demonstrate that torch .compile cannot replicate this fusion (3–7 × slower than our kernels), validating the need for hand-crafted tile-level design. Our fused kernels are also numerically superior, achieving zero mismatches compared to 4.5–11% for cuBLAS.  \n1 Introduction  \nSwiGLU Shazeer [2020] has become the dominant activation function in modern large language models. Qwen-2.5 Yang et al. [2024], LLaMA Touvron et al. [2023], Mistral, and Gemma all employ the gated MLP structure: Gate = A × W1 , Up = A × W2 , Y = SiLU (Gate) ⊙ Up. This pattern requires two independent matrix multiplications followed by an element-wise gated activation, materializing two full intermediate tensors (Gate and Up) in high-bandwidth memory (HBM) between the GeMM and activation stages.  \nAs tensor core compute density increases through quantization (FP8, INT4) and architectural improvements, the relative cost of memory-bound operations grows. We profile the SwiGLU MLPon NVIDIA H100 and find that the activation computation and its associated intermediate tensor materialization consume 9–37% of total MLP execution time depending on model size (Figure 1) . For edge-deployment models (Qwen-2.5 0.5B), SwiGLU accounts for over 30% of MLP time—a substantial overhead that will only worsen as GeMM arithmetic becomes cheaper relative to memory traffic.  \nExisting compiler infrastructure cannot address this bottleneck. PyTorch’s torch .compile with maximum optimization is unable to fuse across two separate GeMMs with different weight matrices—a fundamental limitation of graph-level fusion passes. Our experiments show that torch .compile achieves only 34–94% of eager PyTorch performance for this pattern, and explicit fusion hints provide no meaningful improvement ( \u003C4% change) . This validates the need for hand-crafted, hardware-aware kernel design.  \nFigure 1: SwiGLU activation as a fraction of total MLP execution time across Qwen-2.5 model sizes. Smaller models spend up to 37% of MLP time on SwiGLU and intermediate materialization, motivating tile-level fusion.  \nInspired by FlashAttention’s Dao et al. [2022] success in eliminating intermediate materialization for attention, we apply IO-aware kernel design to the MLP block. However, the MLP fusion challenge is fundamentally different: it involves two independent GeMMs with separate weight matrices that must be coordinated, rather than a single attention computation.  \nWe present two complementary CUTLASS-based SM90 kernels Thakkar et al. [2023] that fuse SwiGLU into GeMM at the tile level:  \n1. First fine-grained GeMM-SwiGLU fusion at the tile level using warp-specialized scheduling—Kernel-1 overlaps Swish computation with Up MMA during t","cbCaic4uVKRkGQDx","https://ap.wps.com/l/cbCaic4uVKRkGQDx","pdf",973373,4,1,15,"English","en",105,"# Abstract\n# Introduction\n## SwiGLU and Gated Activations\n## IO-Aware Kernel Design\n# Related Work\n## SwiGLU and Gated Activations\n## IO-Aware Kernel Design","[{\"question\":\"为什么需要对 SwiGLU 进行 tile-level 融合优化？\",\"answer\":\"SwiGLU 的中间张量（Gate 与 Up）需要在 GeMM 和激活之间物化到 HBM，导致占据 MLP 执行时间的 9–37%。随着 GeMM 计算成本降低，这种内存瓶颈会进一步加重。\"},{\"question\":\"论文提出了哪两类 CUTLASS-based SM90 融合内核？\",\"answer\":\"Kernel-1 在 tile 级别用 warp-specialized 调度把 Swish 计算与 Up-tile loading 进行重叠；Kernel-2 通过自定义 Epilogue Visitor Tree（PairMulStore）把 SwiGLU 与 tile stores 交织，并用 [M,2N] 线程块提升小 batch 的占用率。\"},{\"question\":\"实验结果显示融合带来了哪些收益？\",\"answer\":\"在 NVIDIA H100 上对 Qwen-2.5（0.5B–72B）评测，融合内核相对 PyTorch 最高可达 2.47× 加速，同时将负载从 memory-bound 转向 compute-bound，最高达到 79.5% 峰值 BF16 利用率；数值上实现零不匹配，而 cuBLAS 对比有 4.5–11% 的误差。\"}]",1784189611,38,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"tile-level-activation-overlap-for-efficient-llm-inference","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/tile-level-activation-overlap-for-efficient-llm-inference/83663/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-27","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么需要对 SwiGLU 进行 tile-level 融合优化？","Question",{"text":75,"@type":76},"SwiGLU 的中间张量（Gate 与 Up）需要在 GeMM 和激活之间物化到 HBM，导致占据 MLP 执行时间的 9–37%。随着 GeMM 计算成本降低，这种内存瓶颈会进一步加重。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"论文提出了哪两类 CUTLASS-based SM90 融合内核？",{"text":80,"@type":76},"Kernel-1 在 tile 级别用 warp-specialized 调度把 Swish 计算与 Up-tile loading 进行重叠；Kernel-2 通过自定义 Epilogue Visitor Tree（PairMulStore）把 SwiGLU 与 tile stores 交织，并用 [M,2N] 线程块提升小 batch 的占用率。",{"name":82,"@type":73,"acceptedAnswer":83},"实验结果显示融合带来了哪些收益？",{"text":84,"@type":76},"在 NVIDIA H100 上对 Qwen-2.5（0.5B–72B）评测，融合内核相对 PyTorch 最高可达 2.47× 加速，同时将负载从 memory-bound 转向 compute-bound，最高达到 79.5% 峰值 BF16 利用率；数值上实现零不匹配，而 cuBLAS 对比有 4.5–11% 的误差。","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]