FedACT - Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity

Sep 21, 2026

Uploaded by Asher

Federated Transformer training increasingly uses local AdamW, yet under heterogeneous client data the coordinate-wise reliability of even globally corrected adaptive updates can remain uneven, a problem termed coordinate trust mismatch. Existing federated adaptive optimizers address client-update or communication-round mismatch but still apply corrected adaptive directions uniformly across coordinates. FedACT introduces a global-aware coordinate trust modulation that reallocates coordinate step magnitudes by a coordinate trust score while keeping small non-zero updates. Experiments across federated vision Transformers, CNNs, and LLM pre-training/finetuning show consistent improvements, especially under stronger heterogeneity, and analyses highlight better cross-client direction consistency.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages24

Reading Time1 h 0 min

AccessFree to download

Abstract

Introduction

Adaptive optimization in federated learning

Limitations of existing federated adaptive optimizers

Coordinate trust mismatch

Motivation for coordinate-level modulation

FedACT overview

FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity Shuai Li 1,2,3 , Qinglin Wang 1,2,3 , Ping Luo2,3 , Jiahuan Wang2,3 , Hongyang Hu 1,2,3 , Haotian Mo 1,2,3 Yigui Feng 1,2,3 , Ziang Liu 1,2,3 , Qisong Xiao 1,2,3 , Jie Liu 1,2,3∗, Tao Sun2,3∗ 1Laboratory of Digitizing Software for Frontier Equipment, National University of Defense Technology 2National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology 3 College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China arXiv :2607 .03763v 1 [ cs .LG] 4 Jul 2026 Abstract Federated Transformer training increasingly relies on local AdamW, whose adaptive updates can provide much stronger local progress than SGD-based training. However, under heterogeneous client data, even globally corrected AdamW updates may remain highly uneven in coordinate-wise reliability. We refer to this phenomenon as coordinate trust mismatch. Existing federated adaptive optimizers mainly address mismatch at the client-update or communication-round level, but still apply the corrected adaptive direction densely and uniformly across coordinates. In this paper, we propose FedACT, a global-aware coordinate trust modulation method for federatedAdamW training. FedACT first forms a globally corrected adaptive direction and then reallocates update magnitudes according to a coordinate-wise trust score, assigning larger steps to coordinates jointly supported by local gradients and global correction, while preserving smaller non-zero updates on the remaining coordinates. Extensive experiments on federated vision Transformers, CNNs, LLM pre-training, and LLM finetuning show that FedACT consistently improves over strong federated adaptive baselines, with the largest gains on Transformer models under stronger data heterogeneity. Mechanism analyses further show that FedACT improves cross-client direction consistency, suggesting that coordinate-level trust allocation effectively complements round-level global-local correction. Code will be released. Introduction Adaptive optimization has become central to large-scale distributed training (Kingma and Ba 2014; Loshchilov and Hutter 2017; Zhou et al. 2020; Zhang et al. 2024a; Chen et al. 2023; Jordan et al. 2024), and this trend has naturally extended to federated learning (Reddi et al. 2020; Wang, Lin, and Chen 2022; Sun et al. 2023; Liu et al. 2026, 2025) . In particular, AdamW (Loshchilov and Hutter 2017) is highly effective for Transformer architectures, whose optimization often benefits from heterogeneous effective learning rates rather than a single global learning-rate scale. Recent federated studies (Sun et al. 2023; Liu et al. 2026) further show that local AdamW can substantially outperform local SGD when training vision and language Transformers, motivating ∗Corresponding authors: Jie Liu ([liujie@nudt.edu.cn](liujie@nudt.edu.cn)) and Tao Sun ([suntao.saltfish@outlook.com](suntao.saltfish@outlook.com)) . a shift from purely first-order local training toward adaptive local optimization in federated settings. This trend is also consistent with recent Hessian-based evidence showing that Transformers exhibit pronounced block heterogeneity, which favors coordinate-or block-wise adaptive scaling over single-rate first-order updates (Zhang et al. 2024a,b) . Recent works have begun to address adaptive local optimization in federated learning from the perspective of correction and stabilization (Reddi et al. 2020; Sun et al. 2023; Chen et al. 2025; Liu et al. 2026) . Early adaptive federated optimization methods such as FedOpt (Reddi et al. 2020) introduce server-side adaptivity, while later methods increasingly recognize that heterogeneity-aware correction becomes essential once adaptive optimization is used locally. In particular, FedLADA (Sun et al. 2023) amends local adaptive optimization with a glo

FAQs

What is coordinate trust mismatch in federated AdamW training?

It is the fine-grained reliability imbalance across coordinates where globally corrected adaptive directions can still be unevenly trustworthy under non-IID client data.

How does FedACT address coordinate trust mismatch?

FedACT first constructs a globally corrected adaptive direction, then reallocates update magnitudes per coordinate using a coordinate-wise trust score that leverages both local gradients and global correction.

What benefits does FedACT show over existing federated adaptive baselines?

Experiments on federated vision Transformers, CNNs, and LLM pre-training/finetuning indicate consistent improvements, with the largest gains on Transformer models under stronger data heterogeneity.

Sep 21, 2026

Uploaded by Asher