HCMS: Head-Chunked Multi-Stream Pipeline for Communication-Computation Overlap in Long-Sequence Parallel Attention

Jul 16, 2026

All-to-all sequence parallelism methods run communication and computation strictly in sequence for medium-long inputs, leaving accelerator hardware underutilized. HCMS (Head-Chunked Multi-Stream Pipeline) partitions multi-head attention into head chunks and uses dual CUDA streams to enable fine-grained communication–computation overlap. HCMS works orthogonally with FlashAttention and SDPA, needs no kernel changes, supports uneven partitioning, and preserves numerical equivalence. Experiments on four GPU platforms (2–8 GPUs) show 10%–17.5% speedup vs Ulysses and 5%–14.5% vs Ring Attention for 31K–56K tokens, with 6.8% end-to-end gain on Wan2.2; theory recommends use when communication ratio ρ > 20%.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Abstract

Introduction

Motivation: inefficiency of all-to-all sequence parallelism

Key insight: independence across attention heads

Proposed method: Head-Chunked Multi-Stream Pipeline (HCMS)

Contributions

Experimental validation and results

Theoretical modeling and applicability

HCMS: Head-Chunked Multi-Stream Pipeline for Communication-Computation Overlap in Long-Sequence Parallel Attention Chao Yuan 1 , Pan Li 1 , Yingnan Sun 1 , Jing Liu 1 1 Bilibili Inc. , Shanghai, China {yuanchao, lipan02, sunyingnan, [liujing04](liujing04}@bilibili.com)[}](liujing04}@bilibili.com)[@bilibili.com](liujing04}@bilibili.com) arXiv :2607 .0 18 17v 1 [ cs .DC] 2 Jul 2026 Abstract All-to-all based sequence parallelism methods execute communication and computation strictly in serial when processing medium-long sequences, resulting in hardware resource underutilization. This paper proposes Head-Chunked Multi-Stream Pipeline (HCMS), which exploits the computational independence of multi-head attention by partitioning attention heads into multiple chunks and achieving fine-grained communicationcomputation overlap through dual CUDA streams. HCMS is orthogonally compatible with existing optimizations such as FlashAttention and SDPA, requires no modification to underlying kernels, supports uneven partitioning while maintaining numerical equivalence. Experiments validate the effectiveness across four GPU platforms at 2-8 GPU scales: for typical video generation sequence lengths of 31K-56K tokens, HCMS achieves 10%-17.5% speedup over the Ulysses baseline and 5%-14.5% speedup over Ring Attention; end-to-end acceleration of 6 .8% is achieved on the Wan2 .2 model. Theoretical analysis shows that HCMS benefits are positively correlated with communication ratio ρ, and its use is recommended when ρ > 20% . Keywords: Sequence Parallelism; CommunicationComputation Overlap; Distributed Attention; CUDA Streams; Long Sequence Processing 1 Introduction Large-scale Transformer models [35] have achieved breakthrough advances in natural language processing [6, 3], computer vision [8], and multimodal generation. Video generation represents one of the most prominent application scenarios [12, 2 , 33 , 11] . Taking models such as Sora [25] and Wan2.2 as examples, generating 2-4 second videos at 720P resolution corresponds to sequence lengths of approximately 31K-56K tokens in latent space. This medium-long sequence processing requirement makes sequence parallelism a critical technique—distributing sequences across multiple GPUs to accelerate computation. However, all-to-all based sequence parallelism schemes, represented by DeepSpeed Ulysses [14], exhibit significant efficiency issues: communication and computation execute strictly in serial, causing hardware resource underutilization. We observe that under typical configurations of 4-8 GPUs with PCIe interconnect and 31K-100K token sequence lengths, the communication ratio ρ typically ranges from 15% to 40% . This characteristic provides substantial room for communication optimization—through communication-computation overlap, a theoretical speedup upper bound of 1/(1 − ρ) can be achieved. Existing sequence parallelism methods each have their limitations. Ring Attention [19] employs a ring communication pattern where overlap depends on block-level pipelined execution; when the number of blocks is small, overlap effectiveness is limited, and P − 1 rounds of serial communication are required. Although DeepSpeed Ulysses requires fewer communication rounds, its original implementation executes communication and computation completely in serial, failing to exploit overlap optimization. A fundamental property of multi-head attention is that computations across different heads are mutually independent. This independence implies that the serial dependency of “communication → computation” can be relaxed at the head granularity—rather than waiting for all heads’ input data to complete communication before starting computation, computation for any head can begin immediately once its data is ready. This property provides the theoretical foundation for achieving fine-grained communication-computation overlap atthe head dimension. Based on this insight, we propose the Head-Chunk

FAQs

Why do all-to-all sequence parallelism methods underutilize hardware for medium-long sequences?

Because they execute communication and computation strictly in serial, preventing overlap and leaving GPU resources idle while waiting for communication to finish.

What is the core idea behind HCMS?

HCMS exploits the independence across attention heads by partitioning heads into chunks and scheduling them with fine-grained pipelining using dual CUDA streams, enabling communication–computation overlap at head granularity.

How does HCMS perform and when is it recommended?

Across four GPU platforms and 2–8 GPU scales, HCMS achieves 10%–17.5% speedup over Ulysses and 5%–14.5% over Ring Attention for 31K–56K tokens, with 6.8% end-to-end gain on Wan2.2; theoretical results recommend use when the communication ratio ρ > 20%.

Jul 16, 2026

Upload by Olivia Brown