A Study on Communication Optimization of Distributed Gradient Descent Algorithms Based on Large-Scale Machine Learning - read online free

Aug 3, 2026

Upload by Anda

A distributed training setting faces a rapidly growing gap between the need for frequent gradient synchronization and the limited communication bandwidth in large-scale clusters. The study analyzes bottlenecks of commonly used distributed synchronous stochastic gradient descent and proposes communication-efficient strategies for large-scale machine learning. It compares hybrid gradient compression with the trade-offs of quantization and sparsification, then introduces a Gaussian-averaged stochastic gradient descent design targeting reduced algorithm overhead and achieving constant communication overhead.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages3

Reading Time8 min

AccessFree to download

Introduction

I. Research Status

II. Gradient Compression

(i) Quantification

(ii) Sparsification

(iii) Hybrid Gradient Compression

Gaussian-Averaged Stochastic Gradient Descent

A study on communication optimization of distributed gradient descent algorithms based on large-scale machine learning Hao WU University of Science and Technology of China, Hefei, Anhui 230026 Abstract: In recent years, the rapid development of new generation information technology has resulted in an unprecedented expansion of information capacity. Machine learning algorithms are also increasingly used to compute information sets and build information systems to solve problems whose complexity makes algorithmic solutions infeasible. Examples include autonomous vehicles, speech recognition or user determination (recommendation systems) . The complexity of the machine learning model, combined with the larger amount of data collected, makes it much more expensive to use the model on a single machine, or even impossible to train. Using the computing power of distributed systems is a straightforward, simple solution to the problem. Today, powerful computer clusters are used to train complex deep neural networks on large data sets. However, in large-scale clustered environments, the commonly used distributed synchronous stochastic gradient descent algorithms require frequent node communication to ensure consistency of the gradients (parameters) . This has led to the communication bandwidth being a key constraint for distributed machine learning systems. Keywords: distributed systems; machine learning; distributed optimization; communication eﬃciency Introduction: Gradient compression methods are the most direct way to reduce the communication between nodes, and are mainly divided into gradient quantization and gradient sparsiﬁcation methods. While quantization methods suﬀer from limited compression ratios, sparsiﬁcation methods often lead to serious degradation of model training accuracy. To address these problems, a hybrid gradient compression framework is proposed in this paper after analysing the bottlenecks of the algorithms. The hybrid gradient compression algorithm can combine the advantages of both gradient compression methods, i.e., higher communication compression ratio and lower loss of model training accuracy. The gradient compression method operates on the full gradient value in each iteration, which introduces additional algorithmic overhead. From our research, we found that the complexity of most of the algorithms is above O(n + klogn) . Secondly, although the existing algorithms are able to reduce the communication complexity to a certain extent on the original distributed synchronous stochastic gradient descent algorithm, they are still unable to achieve a constant level of communication overhead of O(1) . To address these problems, we analyse the gradient distribution and variation pattern in model training and propose a Gaussian-averaged stochastic gradient descent algorithm. The algorithm can achieve O(n) algorithm complexity and O(1) communication complexity. I. Research Status With the rapid development of Internet technology, we have entered a brand new era-the era of big data. According to incomplete statistics from online sources, the total scale of Internet data in China has increased at least 50 times during the decade from 2005 to 2015. The speed of development in these areas is far ahead of Moore’s Law for the growth of computing power and Nielsen’s Law for the growth of network bandwidth in the ﬁeld of computer hardware, which we are familiar with. However, due to the increasing volume of information, which is becoming easier to collect, research workers now often need to use collections of millions or even tens of millions of labelled image data to develop image classiﬁers for image recognition tasks (e.g., the ImageNet dataset, which contains 14 million images with more than 20,000 categories, has become an important dataset in the image domain), and also Thousands of hours of speech data sets are used to train speech recognition models (e.g., Baidu has developed and implemented Deep

FAQs

Why does communication become a key bottleneck in distributed synchronous SGD on large clusters?

Frequent node communication is required to ensure gradient/parameter consistency, so communication bandwidth limits overall performance for large-scale distributed machine learning.

How do quantization and sparsification affect training when compressing gradients?

Quantization can limit compression ratios, while sparsification may significantly degrade model training accuracy.

What contribution is made to reduce both algorithmic overhead and communication complexity?

The study proposes a Gaussian-averaged stochastic gradient descent algorithm by analyzing gradient distribution and variation, aiming for O(1) communication complexity and reduced overall computational complexity.

Aug 3, 2026

Upload by Anda