Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum - Research-focused summary

Aug 5, 2026

Upload by Aurelia

Federated Learning (FL) tackles learning from decentralized, privacy-constrained data, yet real-world use is limited by intertwined system and statistical issues, especially under data heterogeneity and partial client participation. Momentum seems beneficial for handling statistical heterogeneity, but existing momentum updates become biased toward recently sampled clients, which can prevent gains over FedAvg. The work introduces Generalized Heavy-Ball Momentum (GHBM), proves convergence under unbounded heterogeneity in cyclic partial participation, and derives adaptive, communication-efficient variants with FedAvg-level complexity. Experiments on vision and language validate improved performance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages36

Reading Time1 h 31 min

AccessFree to download

Abstract

Introduction

Federated Learning overview

Challenges from heterogeneity and partial participation

Prior approaches: control variates and variance reduction

Momentum-based federated learning methods

POLITECNICO DI TORINO Repository ISTITUZIONALE Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum Original Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum / Zaccone, Riccardo; Praneeth Karimireddy, Sai; Masone, Carlo; Ciccone, Marco. -In: TRANSACTIONS ON MACHINE LEARNING RESEARCH. -ISSN 2835-8856. -ELETTRONICO. - (2025) . Availability: This version is available at: 11583/3001458 since: 2025-07-02T08:17:27Z Publisher: [OpenReview.net](OpenReview.net) Published DOI: Terms of use: This article is made available under terms and conditions as specified in the corresponding bibliographic description in the repository Publisher copyright (Article begins on next page) 21 February 2026 Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentum Riccardo Zaccone∗ Politecnico di Torino riccardo.zaccone@polito.it Sai Praneeth Karimireddy USC Viterbi School of Engineering [karimire@usc. edu](karimire@usc. edu) Carlo Masone Politecnico di Torino carlo.masone@polito.it Marco Ciccone Vector Institute [marco. ciccone@vectorinstitute. ai](marco. ciccone@vectorinstitute. ai) Reviewed on OpenReview: [https: // openreview. net/ forum? id= LNoFjcLywb](https: // openreview. net/ forum? id= LNoFjcLywb) Abstract Federated Learning (FL) has emerged as the state-of-the-art approach for learning from decentralized data in privacy-constrained scenarios. However, system and statistical challenges hinder its real-world applicability, requiring efficient learning from edge devices and robustness to data heterogeneity. Despite significant research efforts, existing approaches often degrade severely due to the joint effect of heterogeneity and partial client participation. In particular, while momentum appears as a promising approach for overcoming statistical heterogeneity, in current approaches its update is biased towards the most recently sampled clients. As we show in this work, this is the reason why it fails to outperform FedAvg, preventing its effective use in real-world large-scale scenarios. In this work, we propose a novel Generalized Heavy-Ball Momentum (GHBM) and theoretically prove it enables convergence under unbounded data heterogeneity in cyclic partial participation, thereby advancing the understanding of momentum’s effectiveness in FL. We then introduce adaptive and communication-efficient variants of GHBM that match the communication complexity of FedAvg in settings where clients can be stateful. Extensive experiments on vision and language tasks confirm our theoretical findings, demonstrating that GHBM substantially improves state-of-the-art performance under random uniform client sampling, particularly in large-scale settings with high data heterogeneity and low client participation 1 . 1 Introduction Federated Learning (FL) (McMahan et al., 2017) is a paradigm to learn from decentralized data in which a central server orchestrates an iterative two-step training process that involves 1) local training, potentially on a large number of clients, each with its own private data, and 2) the aggregation of these updated local modelson the server into a single, shared global model. This process is repeated over several communication rounds. While the inherent privacy-preserving nature of FL makes it well-suited for decentralized applications with ∗ Corresponding author 1 Code is available at [https://github.com/RickZack/GHBM](https://github.com/RickZack/GHBM) restricted data sharing, it also introduces significant challenges. Since local data reflects unique characteristics of individual clients, limiting the optimization to a client’s personal data can lead to issues caused by statistical heterogeneity. This becomes particularly problematic when multiple optimization steps are performed before model synchronization, causing clients to drift from the ideal glo

FAQs

What core limitation of existing momentum-based methods is identified in federated learning?

Current momentum updates are biased toward the most recently sampled clients, which can cause performance to fail to surpass FedAvg in practical scenarios.

What does GHBM stand for, and what is its main theoretical contribution?

GHBM is Generalized Heavy-Ball Momentum; it provides a convergence guarantee under unbounded data heterogeneity in cyclic partial participation.

How do the proposed variants of GHBM affect communication and practical applicability?

Adaptive and communication-efficient variants of GHBM are introduced to match FedAvg’s communication complexity in settings where clients can be stateful.

Aug 5, 2026

Upload by Aurelia