Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam

Jul 16, 2026

Upload by Eliana

Local sharpness of the loss and the top Hessian eigenvalue determine the largest stable gradient step, yet typical measurements need Lanczos or Hessian-vector iterations. A single Armijo backtracking line search already encodes this via the accepted step α, which brackets the directional curvature q = g⊤Hg/||g||² within the backtracking band. Across CIFAR-10, Fashion-MNIST, and Imagenette, log α tracks log λ1 with Pearson correlations −0.91 to −0.95, enabling a low-cost online edge-of-stability reading. Used once at initialization, it caps Adam’s learning rate and prevents divergence for initial η between 10⁻³ and 3.0 with ~1% overhead; periodic probing adds no robust benefit.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

Abstract

Introduction

Learning-rate divergence problem in Adam

Prior approaches: range test and adaptive schedules

Proposed Armijo-based initialization probe

Measurement claim: directional curvature and λ1 correlation

arXiv :2607 .03998v2 [ cs .LG] 11 Jul 2026 Directional Curvature from Armijo Backtracking: A Low-Cost Sharpness Probe and a Calibration-Free Learning-Rate Safeguard for Adam Ashmitha R 1 and Jörg Frochte 2 1 Department of Artificial Intelligence and Data Science, Sri Ramakrishna Engineering College, Anna University, Coimbatore 641022, Tamil Nadu, India, [ashmitha.2311011@srec.ac.in](ashmitha.2311011@srec.ac.in) 2 Interdisciplinary Institute for Applied AI and Data Science Ruhr (AKIS), Department of Electrical Engineering and Computer Science, Bochum University of Applied Sciences, Am Hochschulcampus 1, 44801 Bochum, Germany, [joerg.frochte@hs-bochum.de](joerg.frochte@hs-bochum.de) Abstract The local sharpness of the loss, the top Hessian eigenvalue λ 1 , determines the largest stable gradient step, but measuring it normally requires Lanczos or Hessian-vector iterations. We observe that a single Armijo backtracking line search already carries this information at the cost of a few forward passes: the accepted step α brackets the directional curvature q = g ⊤ Hg/∥g∥2 within the multiplicative band set by the backtracking factor. Across CIFAR-10, Fashion-MNISTand Imagenette, log α tracks log λ1 at Pearson −0 .91 to −0 .95, giving a low-cost online Edgeof-Stability reading. Used once at initialisation, this measurement yields a learning-rate cap (a safeguard, not a faster optimiser) that makes Adam robust to a too-large initial learning rate across more than three orders of magnitude (10−3 to 3.0), at about one percent overhead, and it is a no-op when the chosen rate is already safe. One probe is enough: periodic in-training probing adds no robust benefit. The raw-gradient probe exposes the mechanism but needs a safety factor calibrated to the architecture by a one-minute divergence sweep. Probing along Adam’s own update direction removes this calibration: a single fixed safety factor κ = 2 avoids divergence on all nine architectures we test and across the full learning-rate grids of all four benchmarks, and the recipe transfers to AdamW unchanged. 1 Introduction Adam [3] and its variants (AdamW [4], NAdam [5], AMSGrad [6]) are the workhorses of contemporary deep learning. Their per-parameter adaptive scaling absorbs much of the heterogeneity of neuralnetwork gradients and, in practice, decouples optimisation from the worst aspects of architecturespecific gradient geometry. What they do not absorb is the user’s choice of initial learning rate. An η chosen one order of magnitude too large drives the loss to numerical infinity within a handful of steps, well before Adam’s running estimates of the second moment have accumulated enough mass to dampen the update. The result is a single divergent first step, after which the entire training run is wasted. The community has developed several pragmatic responses to this problem. The most principledis the explicit learning-rate range test [8, 9], which trains a model for one or more epochs while sweeping the learning rate and inspects the loss–learning-rate curve for the largest still-stable value. A second line of work removes the learning rate from the user interface entirely by inferring it from running estimates of online-convex-optimisation quantities; D-Adaptation [21], Prodigy [22], Mechanic [23] and the Schedule-Free family [24] are the most recent and most visible representatives. A third, far less elegant but in practice ubiquitous approach is for users to manually iterate over a few candidate values and to discard whatever diverges. None of the three provides a low-cost, calibrate-once safety check that a particular chosen η is safe. We propose a fourth route, designed to coexist with rather than replace any of the above. Before the first Adam update, we run a single Armijo backtracking line search [12] at the randomly initialised parameters: starting from α = 1, we backtrack along the negative initial gradient until the standard sufficient-decrease condi

FAQs

How does a single Armijo backtracking line search help estimate loss sharpness for Adam?

The accepted step size α from Armijo backtracking brackets the directional curvature q = g⊤Hg/||g||² within the multiplicative band induced by the backtracking factor. This provides a sharpness-related estimate without Lanczos or Hessian-vector computations.

What does the method use the measurement for at training time?

At initialization, if the user’s initial learning rate ηinit exceeds κ times the measured stable step, the safeguard silently lowers ηinit to κ·αinit before the first Adam update. Otherwise, it leaves ηinit unchanged, and Adam proceeds unchanged afterward.

Is per-architecture calibration or periodic probing required?

The recommended direction-matched probe removes the need for per-architecture calibration by using a fixed safety factor κ = 2. The authors also report that repeating probing during training provides no robust additional benefit.

Jul 16, 2026

Upload by Eliana