Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data

Jul 16, 2026

Upload by Levi

Adaptive gradient methods can favor max-margin separators different from gradient descent, but a fixed positive numerical stability constant eventually changes the update geometry again. This paper studies the rate-controlled middle case for full-batch linear classification on separable data. For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, it proves normalized iterates converge to a Burg-type convex barrier minimizer over a margin slice. The dynamics are rewritten as entropic mirror ascent on a concave dual, with a KL recursion controlling the dual gap, yielding explicit normalized-iterate envelopes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Abstract

Introduction

Related Work and Problem Setup

Rate-indexed Barrier Path

Contributions and Theorem Scope

Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data Xiangwu Wang1 , Chengwei Cao2 , Yicheng Song3 , Ran Bi1 , Peilin Yu1 1The University of Hong Kong 2University of California, San Diego 3Beijing University of Aeronautics and Astronautics arXiv :2607 .060 13v 1 [ cs .LG] 7 Jul 2026 Abstract Adaptive gradient methods can favor max-margin separators that differ from gradient descent, yet a fixed positive numerical stability constant eventually changes the update geometry again. This paper studies the rate-controlled middle case for full-batch linear classification on separable data. For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice. The proof rewrites the dynamics exactly as entropic mirror ascent on a concave dual objective, controls the dual gap by a KL recursion, and yields an explicit S1/2 normalized-iterate envelope. The static barrier geometry is fully characterized, including KKT conditions and both endpoint limits. Experiments validate the exact dual identities to floating-point error, illustrate the predicted path and rate diagram, and show an empirical fixed-ϵ crossover scaling in cumulative time. We further report robustness and boundary diagnostics for logistic tails, fixed-ϵ crossover, and adaptive-method variants, delineating the scope of the proved smoothed-sign theory. Introduction Implicit bias results for separable classification usually compare endpoints: gradient descent converges in direction to an ℓ2-margin separator, while sign-like, normalized, and adaptive methods are linked to non-Euclidean or ℓ∞ -type geometry (Soudry et al. 2018; Nacson, Srebro, and Soudry 2019; Lyu and Li 2020; Kingma and Ba 2015; Wilson et al. 2017; Zhang, Zou, and Cao 2024; Fan, Schmidt, and Thrampoulidis 2025). The numerical stability constantin Adam-like updates complicates that endpoint story. When the stability term is absent or negligible, the update resembles a sign direction. When it is fixed and positive, sufficiently small gradients eventually see a more gradient-descent-like denominator. The motivating contradiction is that both endpoints are plausible, but neither specifies what separator is selected when the stability term is deliberately annealed at a prescribed exponential rate. We study this question first for the memoryless smoothedsign proxy wt+1 = wt + ηt ~~ ~~|~~ ~~∇∇(Lwt()w|~~ ~~)~~ ~~ϵt , ϵt = ϵ0 exp(−κSt), (1) u2 1.00 0.75 0.50 0.25 0.00 −0.25 −0.50 −0.75 −1.00 Rate-indexed barrier path −1.0 −0.5 0.0 0.5 1.0 u1 0.9 0.5 0.1 κ/ γ∞ Figure 1: Two-dimensional barrier geometry. Contours show the Burg-type barrier, the shaded region is one feasible margin slice, markers are uκ colored by κ/γ∞ , the arrow shows the small-κEuclidean direction, and the star is the ℓ∞ -margin endpoint. The selected dataset is illustrative only and is excluded from aggregate statistics. where St = Pts10 ηs , and absolute values and divisions are coordinatewise. Equation (1) removes Adam’s exponential moving averages while retaining the coordinatewise competition between gradient magnitude and stability. The central object is not a generic transition curve. It is the rate-indexed constrained barrier path shown in Figure 1 . This paper makes four contributions. First, it gives an exact rate-indexed implicit-bias theorem for stability-annealed smoothed-sign descent under weighted exponential loss. Second, it identifies the limiting separator as a Burg-barrier minimizer over a margin slice and characterizes the endpoint geometry. Third, it proves convergence through an exact entropic mirror-ascent representation and a KL recursion. Fourth, it reports theorem-compatible numerical diagnostics, including floating-point dual residuals and a longhorizon showcase, together with boundar

FAQs

What does stability annealing determine in smoothed-sign descent for separable data?

It selects the implicit bias by inducing a rate-indexed constrained barrier path, where the limiting separator is characterized as a Burg-barrier minimizer over a margin slice.

How are the normalized iterates shown to converge?

The paper rewrites the dynamics exactly as entropic mirror ascent on a concave dual objective, then controls the dual gap using a KL recursion to establish convergence to the barrier minimizer.

What is validated through experiments in the study?

Experiments validate the dual identities to floating-point error, illustrate the predicted path and rate diagram, and demonstrate an empirical fixed-ε crossover scaling in cumulative time, along with boundary diagnostics for logistic tails and adaptive-method variants.

Jul 16, 2026

Upload by Levi