No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

Jul 16, 2026

Upload by Liam

Memory-efficient optimizers such as GaLore reduce Adam’s cost by projecting gradients onto a rank-r subspace that is refreshed every T steps, relying on the belief that this subspace is slowly drifting and therefore trackable. Measurements show that beyond a small reproducible core, no such object exists: top-r subspaces estimated from disjoint minibatches disagree as much as estimates refreshed T steps apart. Apparent rotations are dominated by estimator noise, and averaging cannot recover the missing directions, so the low-rank frame is not well defined.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Abstract

Introduction

arXiv :2607 .05872v 1 [ cs .LG] 7 Jul 2026 NO SUBSPACE TO TRACK: NON-IDENTIFIABILITY AND OPTIMIZER STATE IN LOWRANK TRAINING Noel Thomas Mohamed bin Zayed University of Artificial Intelligence [noel.thomas@mbzuai.ac.ae](noel.thomas@mbzuai.ac.ae) ABSTRACT Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace that is recomputed every T steps, on the assumption that this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there is no such object. Two estimates of the top-r subspace computed at the same step from disjoint minibatches disagree as much as estimates computed T steps apart (0 .73 versus 0.74 of the maximal chordal distance √2r, measured at Pythia-160M with r=128): the apparent rotation at every refresh is dominated by estimator noise. This holds across four model families in three architecture classes and from 70M to 6.9B parameters, strengthening with scale, and more weakly in a vision transformer. Only ∼39 of 128 directions are reproducible across minibatches, and averaging cannot recover the rest: under N-fold averaging the gradient’s spectral tail shrinks as N −1/4 rather than the N −1/2 of pure noise, so no averaging budget makes the subspace well defined. At Pythia-1B, a basis averaged over ten thousand steps still moves by ∼84% of the geometric maximum per refresh, and sweeping the averaging strength leaves perplexity essentially unchanged (a 0 .36 perplexity span) . What helps instead follows from treating each refresh as a change of coordinates for Adam’s state. Carrying the second moment blindly is provably about (r−k⋆ )/2 worse than the best rotation-blind estimator, while the first moment transports exactly through the rotation, the optimal linear map under isotropic gradients and the rule LDAdam uses. In a three-seed comparison at 1B and 40k steps, the full LDAdam update reaches 18.7 perplexity at the default β2 =0 .999, beating untransported GaLore after its best β2 fix (19 .3) . Shortening the second-moment memory to β2 =0 .99 helps the refreshing optimizers at their reported recipes, by 1.8 to 26 perplexity, though for canonical GaLore the effect is small and recipe-sensitive; a full-rank control reverses the preference. One measurable fact, subspace non-identifiability, clarifies why GaLore works, which patches work, and what to check before trusting a low-rank assumption: the reproducible rank k⋆ . 1 INTRODUCTION Training large models is increasingly bottlenecked by optimizer memory: Adam stores two moments per parameter, doubling the footprint of the weights themselves. Low-rank gradient optimizers such as GaLore (Zhao et al., 2024) sidestep this by projecting each gradient G ∈ Rm ×n onto the top-r left-singular subspace of G, running Adam in the resulting r-dimensional space, and recomputing the subspace every T steps. This matches full-rank Adam at a fraction of the memory, and a family of refinements has grown around it (Robert et al., 2025; Rajabi et al., 2025b; Zmushko et al., 2024; He et al., 2024; Zhu et al., 2024) . Underlying the whole approach is a geometric premise that is rarely stated and, to our knowledge, was never measured: that the gradient’s top-r subspace is a meaningful, slowly varying object that therefresh tracks. If the premise held, two consecutive estimates of the subspace would nearly coincide, and carrying optimizer state across a refresh would be a small correction. We measure the premise and find that beyond a small reproducible core, there is no such object. The rotation between consecutive subspaces, measured as the chordal distance ∥∆U∥F , saturatesits geometric maximum√2r to 96.7–99.6% at every refresh and at every rank r ∈ {32, ... , 256} during training (Pythia-1B (Biderman et al., 2023); 160M behaves the same): each refresh adopts anearly orthogonal frame. The decisive measurement is the one that removes time. On a pretrained P

FAQs

What is the main premise behind low-rank optimizers like GaLore?

GaLore assumes the gradient’s top-r left-singular subspace is a meaningful, slowly varying object that can be tracked across refresh intervals, enabling Adam state carried between refreshes to remain nearly correct.

Why does the paper argue that there is “no subspace to track”?

Two top-r subspace estimates computed at the same step from disjoint minibatches disagree almost as much as estimates separated by a full refresh interval. The disagreement is attributed to estimator noise rather than true rotation, with only about k*≈39 directions reproducible across minibatches at r=128.

What does the analysis suggest as an alternative explanation for why GaLore can still work?

GaLore works by capturing a roughly constant fraction of gradient energy in a frame that is redrawn each refresh, rather than by tracking a consistent geometric subspace. The paper also proposes treating each refresh as a change of coordinates for Adam’s second-moment state and argues that transporting the state first moment is key.

Jul 16, 2026

Upload by Liam