On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism

Jul 16, 2026

The limited-memory BFGS (L-BFGS) method supports large-scale optimization with low memory and computational cost, but ill-conditioned or non-convex problems can cause the implicit inverse Hessian approximation to develop an exploding condition number, harming numerical stability and convergence. Two-Sided L-BFGS introduces a two-sided geometric envelope that dynamically constrains this condition number. The work proves a uniform condition number bound for all generated approximations and expresses it explicitly via memory depth, problem dimension, and envelope hyperparameters, while preserving curvature information and standard O(mn) complexity. It also establishes asymptotic global convergence under non-convex settings with smoothness and strong Wolfe line-search assumptions, and validates robustness via high-dimensional experiments.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

Introduction

Quasi-Newton methods and motivation

Secant conditions for Hessian and inverse Hessian updates

Safeguarding mechanism for L-BFGS inverse Hessian approximation

Line-search search direction and theoretical results

Numerical experiments and results

arXiv :2607 .05836v1 [math .OC] 7 Jul 2026 Safeguarding Mechanism for L-BFGS Inverse Hessian Approximation Matrix On the Condition Number Upper Bound of the L-BFGS Inverse Hessian Approximation Matrix with a Two-Sided Geometric Envelope Safeguarding Mechanism Don Li [don@pdx.edu](don@pdx.edu) [Department of Mathematics & Statistics](Department of Mathematics & Statistics)[ ](Department of Mathematics & Statistics)Portland State University Portland, OR 97201, USA Abstract The limited-memory BFGS (L-BFGS) algorithm is a cornerstone of large-scale optimization due to its linear memory and computational costs. However, in ill-conditioned or non-convex landscapes, the implicit inverse Hessian approximation can suffer from an exploding condition number, leading to numerical instability and degraded convergence. To address this, we propose Two-Sided L-BFGS, a safeguarded variant that dynamically constrains the condition number of the inverse Hessian operator via a two-sided geometric envelope. Moreover, we show that Two-Sided L-BFGS preserves accumulated curvature information and maintains standard O (mn) memory and per-iteration time complexities. We prove that this geometric envelope yields a uniform bound on the condition number of every inverse Hessian approximation generated by the algorithm. By tracking the algebraic evolution of the extreme eigenvalues through m consecutive quasi-Newton updates starting from a scaled identity matrix, the resulting bound is expressed explicitly as a function of the memory depth, problem dimension, and envelope hyperparameters. Moreover, we show that Two-Sided L-BFGS preserves asymptotic global convergence in non-convex regimes under standard smoothness and strong Wolfe line-search assumptions, matching the theoretical guarantees of L-BFGS variants utilizing the Li-Fukushima cautious update rule. Numerical experiments on high-dimensional optimization problems demonstrate that the proposed method maintains well-conditioned inverse Hessian approximations and improves robustness and convergence behavior on ill-conditioned benchmarks. Keywords: Large-Scale Optimization, L-BFGS Algorithm, Quasi-Newton Methods, Inverse Hessian Approximation, Condition Number Bounding 1 Introduction Quasi-Newton methods are widely used in a plethora of different large-scale optimization settings, ranging from highly non-convex deep learning architectures [Rafati and Marcia (2019)] to numerical weather prediction and climate modeling [Daescu and Navon (2003)] . Such methods balance improved convergence rates with lower computational costs via approximations of the inverse Hessian matrix using only the gradient. Recall that, in general form, Newton’s method employs the update rule, for parameter x k , xk+1 = xk − η[∇2 f (xk)]−1∇f(xk), (1) where [∇2 f (xk)]−1 is the inverse Hessian, ∇f(x k) is the gradient, and η is the learning rate or step size. ∇2 f (xk) provides crucial second-order curvature information and Li guarantees descent for ∇2 f (xk) ≻ 0, but is computationally infeasible in large-scale settings. The Broyden, Fletcher, Goldfarb, and Shanno (BFGS) algorithm was introduced asa Quasi-Newton method to address this computational bottleneck with respect to finding [∇2 f (xk)]−1 . Quasi-Newton methods replace the exact [∇2 f (xk)]−1 with an approximation matrix Hk, where Bk is the approximation matrix of ∇2 f (xk) . As alluded to earlier, as a Quasi-Newton method, BFGS constructs Bk from the gradient, where Hessian approximation updates must satisfy the secant condition, i.e. , Bk+1sk = yk , (2) which we derive as follows: Theorem 1 (Secant Condition for Hessian Approximation Updates) Let f : Rn → R be a twice continuously differentiable function. Let Bk be an approximation of the Hessian matrix ∇2 f (xk) with step vector sk := xk+1 − xk and gradient difference yk :=∇f(xk+1) − ∇f(xk) for consecutive parameter iterates x k , xk+1 . Then any BFGS Hessian approximation update must

FAQs

What problem does the proposed Two-Sided L-BFGS safeguard against?

It prevents the inverse Hessian approximation implicit in L-BFGS from developing an exploding condition number in ill-conditioned or non-convex settings, which would otherwise lead to numerical instability and worse convergence.

How does Two-Sided L-BFGS control the condition number?

It dynamically constrains the inverse Hessian operator’s condition number using a two-sided geometric envelope, applied while preserving accumulated curvature information.

What theoretical guarantees and complexity results are provided?

The method yields a uniform bound on the condition number of every inverse Hessian approximation it generates, with an explicit expression depending on memory depth, problem dimension, and envelope hyperparameters. It also preserves standard O(mn) memory and per-iteration time complexities and maintains asymptotic global convergence under standard smoothness and strong Wolfe line-search assumptions.

Jul 16, 2026

Upload by Theodore