An optimal control approach for neural network architecture adaptation with a posteriori error estimation

Jul 16, 2026

A novel method adapts neural network architecture along network depth using a posteriori error estimation. Neural network training is formulated as a continuous-time optimal control problem, yielding rigorous layer-wise approximation error estimates. The error decomposition supports a principled depth refinement strategy by inserting new layers at positions of maximum estimated error to capture nonlinear solution structure. We derive computable upper bounds via dual weighted residual methodology and provide explicit interval-wise error bounds for targeted architecture refinement. Experiments on scientific datasets, including learning the observable-to-parameter map for Navier-Stokes, show improved generalization over existing adaptation methods.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

Introduction

Neural network depth as a key factor

Two families of automated architecture selection

Prior work on growing width vs growing depth

Missing quantitative link and the paper’s contribution

Crossmark RECEIVED dd Month yyyy REVISED dd Month yyyy arXiv :2607 .07637v 1 [ cs .LG] 8 Jul 2026 PAPER An optimal control approach for neural network architecture adaptation with a posteriori error estimation C G Krishnanunni 1 , Thomas Scott 1 , Tan Bui-Thanh 1 ,∗ 1 Department of Aerospace Engineering & Engineering Mechanics, UT Austin ∗ Oden Institute for Computational Engineering and Sciences, UT Austin. E-mail: [krishnanunni@utexas.edu](krishnanunni@utexas.edu) Keywords: Neural architecture adaptation, A posteriori error estimation, Optimal control theory. Abstract This work presents a novel approach for adapting neural network architecture along the depth based on a posteriori error estimation. By formulating neural network training as a continuous-time optimal control problem, we derive rigorous error estimates that quantify how approximation error distributes across network layers. This error decomposition enables a principled depth adaptation strategy: new layers are inserted at locations of maximum estimated error, allowing the network to efficiently capture complex, nonlinear variations in the underlying problem. Our framework introduces a novel network architecture that treats weights and biases as piecewise linear functions varying across layers, with the error estimator bounding the discrepancy between this discrete representation and the true continuous optimal control solution. The approach leverages dual weighted residual methodology from finite element analysis to derive computable upper bounds on the functional error. A key theoretical contribution is the derivation of explicit error bounds that decompose the total approximation error into interval-wise contributions, providing a rigorous basis for targeted architecture refinement. We demonstrate the effectiveness of our method on scientific datasets, including learning the observable-to-parameter map for the Navier-Stokes equation. Numerical results reveal that our approach consistently outperforms existing architecture adaptation methods in terms of generalization performance. 1 Introduction Depth of a neural network plays a key role in the observed empirical success of deep learning. Stacking layers lets a network build successively more abstract features out of raw input data [1, 2] . Several of the architectures responsible for the largest jumps in benchmark performance over the past decade differ from their predecessors mainly in how many layers they contain [3, 4] . A key question is how many layers does a given task actually require, and how should the available parameters be distributed among them? In practice this question is still settled almost entirely by trial and error ora metaheuristic optimization-based search procedure [5, 6] . A network with too little depth and small width may be unable to represent the target function at all, whereas an unnecessarily deep network wastes compute and is more prone to overfitting. A more systematic, less wasteful approach to setting network depth therefore has clear practical value. Two broad families of methods have been proposed to automate this choice. The first treats architecture selection as a search problem. A population or sequence of candidate networks is proposed, evaluated using evolutionary computation [5, 6], reinforcement learning [7], or randomized search [8], and the best candidate is retained. These neural architecture search (NAS) methods can produce excellent architectures, but doing so typically requires training large numbers of candidates which makes the overall procedure costly, particularly when a single training run is itself expensive. The second family avoids searching a discrete space of architectures altogether. Instead, a single small network is trained and then grown, with new parameters added as training proceeds rather than fixed in advance [9, 10 , 11 , 12 , 13] . Within this second family, the bulk of prior work targets towards growing t

FAQs

How does the approach adapt neural network depth during training?

Training is reformulated as a continuous-time optimal control problem, and posteriori error estimates determine where to insert new layers so added capacity targets the largest estimated approximation errors.

What is the role of dual weighted residual methodology in this framework?

It is used to derive computable upper bounds on the functional (approximation) error, enabling practical estimation rather than only qualitative guidance.

What datasets and task are used to demonstrate the method?

The method is evaluated on scientific datasets, including learning the observable-to-parameter map for the Navier-Stokes equation, where it improves generalization compared with existing architecture adaptation methods.

Jul 16, 2026

Upload by Evangeline