SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference

Jul 16, 2026

Upload by Rowan

Reasoning models and agentic systems make per-token latency a critical bottleneck for LLM inference, especially because intermediate reasoning tokens are not consumed by humans and minimal batching is used. While tensor parallelism sharding improves memory bandwidth, scaling increases All-Reduce overhead that can dominate token generation time. Removing All-Reduce boosts token throughput by 43% on Llama-3.1-8B with 8 H200 GPUs. SiFAR targets All-Reduce synchronization costs, eliminating barrier overheads with dual buffering, improving scalability via redundant switch-side reduction, and reducing top barriers using speculative reduction, cutting All-Reduce latency up to 52%.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Introduction

The Low-Latency Challenge

All-Reduce: The Emerging Bottleneck

Limitations of Prior Work

All-Reduce Algorithms

SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference * Hritvik Taneja [htaneja3@gatech.edu](htaneja3@gatech.edu) * Anish Saxena [asaxena317@gatech.edu](asaxena317@gatech.edu) Abhishek Revinipati* [arevinipati3@gatech.edu](arevinipati3@gatech.edu) Jae Hyung Ju* Neal C. Crago† Moinuddin Qureshi* [jju35@gatech.edu](jju35@gatech.edu) [ncrago@nvidia.com](ncrago@nvidia.com) [moin@gatech.edu](moin@gatech.edu) arXiv :2607 .08973v 1 [ cs .DC] 9 Jul 2026 Abstract The rise of reasoning models and agentic systems has made LLMtoken-generation latency a key bottleneck. Unlike chatbots, where latency gains saturate at human reading speed, these systems generate intermediate reasoning tokens that are not consumed by humans. As a result, per-token latency directly determines end-to-end response time. To reduce latency, inference engines operate with minimal batching, making token generation bandwidth-bound. Tensor Parallelism addresses this bottleneck by sharding model weights across GPUs and loading them in parallel. However, scaling to more GPUs introduces All-Reduce overheads that grow with GPU count. We observe that removing All-Reduce improves token throughput by 43% for Llama-3.1-8B on 8 H200 GPUs. To address this overhead, we propose Synchronization-Free AllReduce (SiFAR), which reduces synchronization overheads incurred by All-Reduce during low-latency inference. Existing oneshot and twoshot algorithms incur high synchronization overheads due to barriers before and after communication, accounting for 32–62% of the latency for small payloads we observe in low-batch serving. To tackle this, first, we find that the bottom barrier in oneshot enforces a WAW dependency, and eliminate it by co-designing communication and model execution to enable dual buffering. However, oneshot scales poorly with GPU count. Twoshot performs better at higher TP degrees, but incurs an unavoidable bottom barrier. To overcome this, we leverage in-switch reduction in modern switches, primarily used for reduce-scatter. We propose redundant pull, where each GPU reduces the entire all-reduce payload at the switch. This improves oneshot scalability while retaining its no bottom barrier advantage. Finally, to reduce top barrier overhead, we observe that each token generation step issues multiple All-Reduce operations, keeping GPUs tightly synchronized after the first All-Reduce. We therefore propose speculative reduction, which initiates data transfer without waiting for the top barrier and ensures correctness via a lightweight validation mechanism. SiFAR reduces All-Reduce latency by up to 52% and improves end-to-end throughput by 18.6% for Llama-3.1-8B and 13.1% for Qwen3.5-397B-17B at TP=8 . Keywords Low-Latency LLM Inference, All-Reduce, In-Switch Reduction 1 Introduction The advent of agentic systems [1, 26, 31] and reasoning models [17] has made low-latency LLM token generation critical. In humanfacing LLM applications such as chatbots, each token is read by users, so reducing latency beyond human reading speed provides diminishing returns. In contrast, these systems generate intermediate reasoning tokens that are not consumed by humans, so per-token ∗ Georgia Institute of Technology, Atlanta, Georgia, USA †NVIDIA, Santa Clara, California, USA latency compounds across steps and directly impacts end-to-end response time. Recent proposals for hardware accelerators tailored for reasoning [2] and industry platforms like NVIDIA Groq LPU [39] further emphasize the importance of optimizing for low-latency inference. In this work, we focus on reducing token generation latency in GPU-based inference. The Low-Latency Challenge. The first step towards reducing token generation latency is to minimize batching, since larger batches delay individual token generation by (1) increasing the size of KV Cache and (2) activating more model parameters (for MoE [29]) . With minimal batching, loading model weights and KV Cache from H

FAQs

Why does per-token latency become especially important in reasoning and agentic LLM systems?

These systems generate intermediate reasoning tokens over multiple steps, so latency compounds across generation steps and directly affects end-to-end response time.

How do tensor parallelism approaches help with the low-batch memory bandwidth bottleneck, and what new issue appears?

Tensor parallelism partitions model weights and KV cache across GPUs and loads in parallel to improve bandwidth, but scaling adds communication and synchronization overheads from All-Reduce.

What is SiFAR, and what bottlenecks in All-Reduce does it target?

SiFAR is a synchronization-free All-Reduce approach that reduces All-Reduce overheads caused by synchronization barriers during low-latency inference, improving throughput and reducing end-to-end token generation latency.

Jul 16, 2026

Upload by Rowan