CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems

Jul 16, 2026

Upload by Riley

CTA-pipelining introduces a latency-oriented execution paradigm for tightly coupled shared-memory multi-GPU systems, targeting large language model serving where performance is constrained by single-batch input latency rather than throughput. Instead of treating coherent interconnects as mere high-speed networks, the method exploits Cooperative Thread Array (CTA) level dependencies to run dependent kernels concurrently across GPUs. Implemented with CUTLASS, cuBLAS, and NCCL on 8-GPU H200/B200, it reduces latency up to 31.8% versus microbatching and 29.6% versus tensor parallelism.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

Introduction

Multi-GPU serving objectives and latency constraints

Hybrid parallelism baselines (PP and TP)

Limitations and gap: intra-layer inter-operator space

CTA-pipelining approach and integration strategy

CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems Tingkai Liu∗†, Muralidhar Andoorveedu†, Sanjoy Das†, Sanjay Patel†, Volodymyr Kindratenko∗ ∗ University of Illinois Urbana-Champaign † NVIDIA Corporation {tingkai2, [kindrtnk](kindrtnk}@illinois.edu)[}](kindrtnk}@illinois.edu)[@illinois.edu](kindrtnk}@illinois.edu), {tingkail, mandoorveedu, sanjoyd, [sanjpatel](sanjpatel}@nvidia.com)[}](sanjpatel}@nvidia.com)[@nvidia.com](sanjpatel}@nvidia.com) arXiv :2607 .07862v 1 [ cs .DC] 8 Jul 2026 Abstract—The evolution of compute infrastructure has transformed multi-GPU systems into tightly integrated sharedmemory structures. However, current software still mostly treats these coherent interconnects simply as high-speed networks. Simultaneously, the demand for serving Large Language Models under latency constraints has shifted GPU workload optimization from being throughput-driven to latency-bound, necessitating latency-oriented scaling methods beyond Tensor Parallelism (TP). Thus, we introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems. As a latency-oriented spatial scaling technique, CTA-pipelining leverages dependencies at the Cooperative Thread Array level, enabling concurrent execution of dependent kernels across GPUs. We demonstrate its capability using CUTLASS, cuBLAS, and NCCL libraries on 8-GPU H200 and B200 systems. Results show on 2-layer GEMM, representing the MLP operation, CTApipelining reduces latency by up to 31.8% compared to microbatching, and 29.6% compared to TP. It can also be combined with TP as an orthogonal scaling dimension to further push the latency boundary. Index Terms—Graphics Processing Unit, Parallel programming, Distributed Computing, Machine Learning. I. INTRODUCTION Since the development of the Transformer architecture [1],[2], optimizing Large Language Model (LLM) inference for production has become a critical challenge [3] . Modern serving frameworks have two main objectives: maintaining high aggregate throughput for cost efficiency, and meeting latency Service-Level Objectives [4] . In highly interactive scenarios, the limiting factor becomes single-batch user input latency. Since GPUs have traditionally been designed as throughputoriented devices, minimizing latency for a single-batch request introduces new system-level requirements. Correspondingly, to support the rapid scaling of LLM workloads, modern multi-GPU hardware systems have evolved into tightly coupled architectures, calling for novel software paradigm to fully exploit them. Systems such as the NVIDIA GB200 NVL72 [5] utilize NVLink and NVSwitch [6] interconnects not only to provide high peer-to-peer bandwidth, but also to enable the multi-GPU cluster to function with a unified shared-memory space. Although recent advancements have introduced sophisticated serving frameworks to maximize multi-GPU deployment efficiency [7]–[11], as well as efforts to build systematic abstractions for programming multi-GPU workloads [12], [13], there remains untapped potential for novel execution paradigms that natively exploit these tightly coupled clusters as holistic shared-memory systems. At the current stage, the standard paradigm for largescale LLM deployment on multi-GPU systems relies on hybrid parallelism strategies [14]–[17], primarily combining Pipeline Parallelism (PP) [18], [19] and Tensor Parallelism (TP) [20] . While emerging techniques such as Expert Parallelism (EP) [21] and disaggregated serving [22] offer further optimizations, they are highly workload-specific. Therefore, PP and TP remain the universal baselines. PP mainly focuses on improving overall serving throughput by operating at the inter-layer level to distribute transformer blocks across devices. TP provides both throughput improvement and latency reduction by spatially sharding computations at the operator level. However, TP introduces additional collective communication (e.g., All

FAQs

What problem does CTA-pipelining address in multi-GPU LLM serving?

It targets latency constraints in interactive LLM serving, where single-batch user input latency becomes the bottleneck rather than aggregate throughput.

How is CTA-pipelining different from existing pipeline and tensor parallelism methods?

It performs latency-oriented spatial scaling inside the intra-layer, inter-operator execution space by exploiting CTA-level dependencies to execute dependent kernels concurrently across GPUs.

What performance gains are reported, and with which libraries and hardware?

On 8-GPU H200 and B200 systems, CTA-pipelining reduces latency up to 31.8% versus microbatching and 29.6% versus tensor parallelism, using CUTLASS, cuBLAS, and NCCL for evaluation on 2-layer GEMM (MLP).

Jul 16, 2026

Upload by Riley