GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining

Jul 16, 2026

Upload by Miles

Gradient communication is a primary scaling bottleneck in large language model pretraining. Lowering gradient precision (e.g., FP8, NVFP4) reduces communication volume, but Euclidean-space quantization can harm performance due to direction-dependent distortions from highly anisotropic gradients. GIFT introduces geometry-informed gradient scaling that quantizes in geometry-aware coordinates, transforming gradients into a near-isotropic space before FP8 communication. A simplified geometry-aware transform uses low-rank approximation and selective application to balance overhead. Experiments on Llama-300M/600M show faster pretraining and better downstream preservation than direct Euclidean FP8.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Gradient communication bottlenecks

Related low-precision gradient methods

Motivation for geometry-aware scaling

Method Overview

GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining Jieying Wang 1 , Shuyuan Fan2 , Mingkai Zheng2 , and Zhao Zhang2 1Department of Computer Science, Rutgers University, Piscataway, NJ, USA 2Department of Electrical and Computer Engineering, Rutgers University, Piscataway, NJ, USA arXiv :2607 .07494v 1 [ cs .DC] 8 Jul 2026 Abstract—Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can significantly reduce the communication volume. Existing methods quantize gradients via linear or nonlinear mappingsin Euclidean space, often degrading model performance because highly anisotropic gradients incur direction-dependent distortion. We present GIFT, a geometry-informed gradient scaling method that performs low-precision communication in geometry-aware coordinates. By transforming gradients into a near-isotropic space before quantization, GIFT makes low-precision representations substantially more faithful to their high-precision counterparts. GIFT only changes the coordinate system used for low-precision gradient communication and does not change the optimizer, training recipe, communication collective, or lowprecision format. We also develop a simplified geometry-aware transformation algorithm with low-rank approximation and selective application to balance the computation overhead and communication reduction. We examine the empirical convergence of GIFT using Llama-300M and Llama-600M models. Our results show that GIFT reduces the end-to-end pretraining time of Llama-600M by 7.6% on 64 NVIDIA GH200 Superchips, while improving the downstream task preservation profile over direct Euclidean FP8 communication under the same optimizer and communication path. Index Terms—large language models, distributed pretraining, gradient communication, communication compression, lowprecision communication, geometry-aware communication, KFAC I. INTRODUCTION Gradient communication is a major scaling bottleneck in distributed LLM pretraining [1] . In data-parallel, 3D-parallel, or more complex parallel pretraining strategies, the optimizer needs to communicate gradients across graphics processing units (GPUs) . Standard communication libraries, such as NCCL and MPI, implement the ring allreduce algorithm [2], whose cost depends on the model size and job size (i.e., the number of GPUs) . Previous work reports that communication accounts for 40% of the overall pretraining time for GPT-8.3Bpretraining across 128 NVIDIA A100 GPUs [3] . Reducing communication volume via low-precision gradients is an effective way to lower communication overhead, asthe bits saved for each gradient representation directly translate into proportional reductions in bandwidth consumption. Existing low-precision training techniques, such as FP8-LM [4] and COAT [5], mainly apply scaling and quantization in Euclidean tensor coordinates, which can still lead to degraded model performance. They are designed specifically for the AdamW optimizer [6] . FP8-LM uses FP8 format for model weights, activations, gradients, and first moment, while the second moment is stored in 16-bit format for division stability. COAT uses dynamic range strategies (i.e., SX = ~~ ~~amF~~3~~XP~~32~~)) and mixed-granularity activation quantization to quantize second moment and activation, while the gradients are still in 16-bit format for training stability. SDP4Bit [7] leverages the sharded data parallel model placement and implements 4-bit gradient communication via Fourier transform and hierarchical allto-all communication. However, neither FP8-LM nor COAT achieves comparable model performance to the BF16 baseline. FP8-LM shows that seven out often downstream tasks perform worse than the baseline. COAT has only one with a higher score than the baseline among the four downstream tasks. SDP4Bit only evaluates model performance using validation

FAQs

Why is gradient communication a bottleneck in distributed LLM pretraining?

In data-parallel or more complex parallel strategies, the optimizer must exchange gradients across GPUs using ring allreduce, whose cost grows with model and job size. Prior reports show communication can consume a large fraction of total pretraining time.

What limitation affects existing low-precision gradient quantization methods?

Most methods quantize gradients in Euclidean tensor coordinates using linear or nonlinear mappings, which can degrade performance. Highly anisotropic gradients can be distorted in a direction-dependent way, causing the synchronized updates to drift from high-precision behavior.

How does GIFT improve fidelity and efficiency for FP8 gradient communication?

GIFT changes only the coordinate system used for low-precision gradient communication. It transforms gradients into a near-isotropic, geometry-aware space before quantization, so FP8 scaling introduces more comparable errors across dimensions, improving downstream task preservation while reducing pretraining time.

Jul 16, 2026

Upload by Miles