An Additively Preconditioned Trust Region Strategy for Machine Learning

Sep 17, 2026

Uploaded by Skyler

Modern machine learning training, especially for deep neural networks, relies on solving large-scale highly nonconvex optimization problems with rough objective landscapes. Motivated by parallel preconditioners for Krylov methods in large linear systems, this work presents a nonlinearly preconditioned trust-region method using an additive Schwarz correction each minimization step. The proposed APTS combines right-preconditioned additive Schwarz with a classical trust-region scheme, solving local nonlinear subproblems in parallel and assembling corrections additively to accelerate convergence while reducing hyperparameter sensitivity.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Abstract

Introduction

arXiv :2512 . 14286v1 [math .NA] 16 Dec 2025 An Additively Preconditioned Trust Region Strategy for Machine Learning Samuel Cruz Alegría* Bindi Çapriqi† Shega Likaj 1,2 Ken Trotti2 Rolf Krause 1,2 1Università della Svizzera Italiana 2 King Abdullah University of Science and Technology Abstract Modern machine learning, especially the training of deep neural networks, depends on solving largescale, highly nonconvex optimization problems, whose objective function exhibit a rough landscape. Motivated by the success of parallel preconditioners in the context of Krylov methods for large scale linear systems, we introduce a novel nonlinearly preconditioned Trust-Region method that makes use of an additive Schwarz correction at each minimization step, thereby accelerating convergence. More precisely, we propose a variant of the Additively Preconditioned Trust-Region Strategy (APTS), which combines a right-preconditioned additive Schwarz framework with a classical Trust-Region algorithm. By decomposing the parameter space into sub-domains, APTS solves local non-linear subproblems in parallel and assembles their corrections additively. The resulting method not only shows fast convergence; due to the underlying Trust-Region strategy, it furthermore largely obviates the need for hyperparameter tuning. 1 Introduction Given a dataset {(x(i), y (i))}, with x(i) ∈ Rq and y (i) ∈ Rp, in supervised learning parameters θ ∈ Rnare to be identified such that the following minimization problem is approximately solved: M inRn f (θ) := in1Rn M X ℓ 􀀀N(θ;x(i)), y (i)􀀁 . (1) i=1 Here f : Rn −→ R is the empirical risk or average loss over the whole dataset, ℓ : Rp × Rp −→ R denotes the per-sample loss and N(θ,·) : Rq −→ Rp is the forward pass of a Neural Network (NN), that mapsan input vector x ∈ Rq to a prediction y ∈ Rp under the model’s parameters θ . In the context of NNs, approximately solving (1) is known as training. Loss functions in Machine Learning are typically large scale, i.e. defined on large data-sets, and are highly nonconvex with varying curvature across different layers or parameter blocks. They often admit for many saddle points and poor local minima, i.e. minima with insufficient approximation properties of the resulting network. Moreover, training usually starts from random or heuristic initializations, which can lie far from a “good” solution. Under these conditions, naïve gradient updates may stall or converge very *Università della Svizzera Italiana †King Abdullah University of Science and Technology. Corresponding author: [bindi.capriqi@kaust.edu.sa](bindi.capriqi@kaust.edu.sa) slowly, on extended plateaus around high-dimensional saddle points [13, 16] . Therefore, ensuring stable, hyperparameter-robust descent from arbitrary starting points remains a central challenge for scalable ML optimization and motivates our exploration of additive preconditioned Trust Region (TR) methods. The most widely used optimization methods for problem (1) are Stochastic Gradient Descent (SGD) and its variants [27, 31, 44], most notably Adaptive Moment estimation (Adam) [18] . Further, the limitedmemory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) quasi-Newton method is particularly effective for ill-conditioned, large-scale problems, since it builds an efficient low-rank approximation of the inverse Hessian matrix [23] . Although these methods are widely used and have wide-ranging benefits, they also have their specific drawbacks. For SGD, each iteration is inexpensive, but an asymptotically diminishing learning-rate schedule is required to guarantee convergence even in convex settings [27] . Variance-reduced variants such as SVRG [33], SAGA [2], and SARAH [29] alleviate this by controlling the gradient noise and can attain linear convergence without vanishing stepsizes for finite-sum objectives, yet they still incur periodic full-gradient costs and their gains for deep, highly nonconvex networks are mixed [3] .

FAQs

What problem does the document address in machine learning optimization?

It addresses training deep neural networks as a large-scale, highly nonconvex optimization task with rough landscapes, saddle points, and poor local minima where naive gradient updates can stall.

How does the proposed method modify a trust-region approach?

It introduces a right-preconditioned additive Schwarz framework together with a classical trust-region algorithm, applying an additive Schwarz correction at each minimization step.

What benefits does the method claim regarding convergence and hyperparameter tuning?

It reports fast convergence and, due to the trust-region strategy, largely reduces the need for hyperparameter tuning.

Sep 17, 2026

Uploaded by Skyler

An Additively Preconditioned Trust Region Strategy for Machine Learning_Page 1

Open in WPS to view all 13 pages