From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators

Jul 16, 2026

Upload by Finn

Derives an asymptotic risk characterization for regularized empirical risk minimization estimators whose tuning parameters are selected via n-fold cross-validation. The out-of-sample prediction loss converges in distribution to squared-error risk of corresponding shrinkage estimators in the normal means model, with tuning driven by Stein’s unbiased risk estimate (SURE). The resulting risk function yields a parameter-dependent view of predictive performance beyond worst-case regret bounds, and is built on uniform CV-to-SURE convergence plus generic separation of SURE’s global minimum.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages70

Reading Time2 h 56 min

Abstract

Introduction

Background and problem setup

Risk functions and motivation

Main result

Key steps

arXiv :2603 .20388v2 [math . ST] 10 Jul 2026 From Cross-Validation to SURE: Asymptotic Risk of Tuned Regularized Estimators Karun Adusumilli∗ Maximilian Kasy† Ashia Wilson‡ July 13, 2026 Abstract We derive the asymptotic risk function of regularized empirical risk minimization (ERM) estimators tuned by n-fold cross-validation (CV) . The out-of-sample prediction loss of such estimators converges in distribution to the squared-error loss (risk function) of shrinkage estimatorsin the normal means model, tuned by Stein’s unbiased risk estimate (SURE) . This risk function provides a more fine-grained picture of predictive performance than uniform bounds on worst-case regret, which are common in learning theory: it quantifies how risk varies with the true parameter. As key intermediate steps, we show that (i) n-fold CV converges uniformly to SURE, and (ii) while SURE typically has multiple local minima, its global minimum is generically well separated. Well-separation ensures that uniform convergence of CV to SURE translates into convergence of the tuning parameter chosen by CV to that chosen by SURE. ∗ Department of Economics, University [of Pennsylvania. akarun@sas.upenn.edu](of Pennsylvania. akarun@sas.upenn.edu). †Corresponding author. Department of Economics, University of Oxford. maximil[ian.kasy@economics.ox.ac.uk. Maximilian Kasy was](ian.kasy@economics.ox.ac.uk. Maximilian Kasy was) supported by the Alfred P. Sloan Foundation, under the grant “Social foundations for statistics and machine learning.” ‡Department of Electrical Engineering and Computer Science, MIT. 1 Introduction Background The goal of supervised learning is to produce good predictions for new observations.1 An important class of estimators for supervised learning are regularized empirical risk minimization (ERM) estimators that are tuned using cross-validation (CV) . ERM estimators minimize in-sample average prediction loss (empirical risk), among a given class of predictors. Examples include ordinary least squares and maximum likelihood. ERM estimators are prone to overfitting if the class of predictors is large. Such estimators achieve low in-sample loss, but can perform poorly for new observations. To counter overfitting, regularization is used. Regularization adds a penalty term to the ERM objective; common penalties include the L2 norm of parameters (in Ridge regression) and the L 1 norm (in Lasso regression) . Adding a penalty avoids overfitting, by reducing the estimator variance, at the cost of introducing some bias, which might result in underfitting. To achieve good performance, avoiding both overfitting and underfitting, the amount of penalization needs to be carefully tuned. This can be done by choosing weights for the penalty that minimize a cross-validation estimate of predictive loss. We focus on n-fold CV, where predictions are evaluated for one hold-out observation at a time, and predictive loss is estimated by averaging evaluations over each of the n observations. Risk functions The present paper characterizes the behavior of estimators of this form by deriving an asymptotic approximation to their risk function. A large literature in learning theory characterizes such estimators by proving bounds on their worst-case regret—the supremum over data-generating processes (DGPs) of the difference between an estimator’s risk and the expected loss of the best predictor in the given class. Such bounds provide strong robustness guarantees, but they might not be informative about the behavior of predictive algorithms for realistic DGPs. By focusing on the risk function, 1We would like to dedicate this paper to Gary Chamberlain, whose conversations provided the original inspiration for this project. Figure 1: Risk function for JS-shrinkage, dimension 10 1 2 3 4 5 6 ∥θ∥ we obtain a more fine-grained characterization: the risk function tells us how expected predictive performance depends on the DGP, while worst-case

FAQs

What does the paper derive about cross-validation-tuned regularized estimators?

It derives an asymptotic risk function, showing that the out-of-sample prediction loss of n-fold CV-tuned regularized ERM estimators converges in distribution to the squared-error loss of a corresponding SURE-tuned shrinkage estimator in the normal means model.

How is SURE connected to the tuning performed by cross-validation?

The paper proves that n-fold cross-validation converges uniformly to SURE and that the tuning parameter chosen by CV converges to the tuning parameter chosen by SURE, relying on well-separation of SURE’s global minimum.

Why do the authors focus on risk functions instead of worst-case regret bounds?

Risk functions provide a finer, parameter-dependent characterization of expected predictive performance under realistic data-generating processes, whereas worst-case regret only describes behavior under the least favorable processes.

Jul 16, 2026

Upload by Finn