Selective machine learning of doubly robust functionals - Research paper overview

Aug 3, 2026

Upload by Ophelia

Selective machine learning addresses model selection in semiparametric settings where nuisance parameters may be high-dimensional and the target is a finite-dimensional functional. The work introduces two model selection criteria that reduce bias by using a new pseudo-risk definition tailored to doubly robust estimating functions. A multi-fold cross-validation version is shown to have an oracle property, nearly matching an oracle with knowledge of each candidate model’s pseudo-risk, and a smooth approximation enables valid post-selection inference. The method is applied to selecting semiparametric models for average treatment effect estimation using ensembles of machine learners to handle confounding in observational studies.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages54

Reading Time2 h 16 min

AccessFree to download

Introduction

Doubly robust inference and motivating challenges

Proposed selective model selection framework

Pseudo-risk criteria and oracle results

Cross-validation and post-selection inference

Application to average treatment effect estimation

View metadata, citation and similar [papers at ](papers at core.ac.uk)[core.ac.uk](papers at core.ac.uk) brought to you by CORE [provided by](provided by arXiv.org)[ arXiv.org](provided by arXiv.org) e-Print Archive arXiv : 19 11 .02029v2 [ stat .ME] 29 Apr 2020 Selective machine learning of doubly robust functionals Yifan Cui, Eric Tchetgen Tchetgen Abstract While model selection is a well-studied topic in parametric and nonparametric regression or density estimation, model selection of possibly high-dimensional nuisance parameters in semiparametric problems is far less developed. In this paper, we propose a selective machine learning framework for making inferences about a 􀀌nite-dimensional functional de􀀌ned on a semiparametric model, when the latter admits a doubly robust estimating function. We introduce two model selection criteria for bias reduction of functional of interest, each based on a novel de􀀌nition of pseudo-risk for the functional that embodies this double robustness property and thus may be used to select the candidate model that is nearest to ful􀀌lling this property even when all models are wrong. We establish an oracle property for a multi-fold cross-validation version of the new model selection criteria which states that our empirical criteria perform nearly as well as an oracle with a priori knowledge of the pseudo-risk for each candidate model. We also describe a smooth approximation to the selection criteria which allows for valid post-selection inference. Finally, we apply the approach to model selection of a semiparametric estimator of average treatment e􀀋ect given an ensemble of candidate machine learners to account for confounding inan observational study. keywords Machine Learning, Doubly Robust, In􀀍uence Function, Model Selection, Average Treatment E􀀋ect 1 Introduction Model selection is a well-studied topic in statistics, econometrics and machine learning. In fact, methods for model selection and corresponding theory abound in these disciplines, although primarily in settings of parametric and nonparametric regression and density estimation. Model selection methods are far less developed in settings where one aims to make inferences about a 􀀌nite dimensional, pathwise di􀀋erentiable functional de􀀌ned on a semiparametric model. Model selection for the purpose of estimating such a functional may involve selection of an in􀀌nite dimensional parameter, say a nonparametric regression for the purpose of more accurate estimation of functional in view, which can be considerably more challenging than selecting a regression model strictly for the purpose of prediction. This is because whereas the latter admitsa risk, e.g., mean squared error loss, that can be estimated unbiasedly and therefore can be minimized with small error, the risk of a semiparametric functional will typically not admit an unbiased estimator and therefore may not be minimized without excessive error. This is an important gap in both model selection and semiparametric theory which this paper aims to address. Speci􀀌cally, we propose a novel approach for model selection of a functional de-􀀌ned on a semiparametric model, in settings where inferences about the targeted functional involves in􀀌nite dimensional nuisance parameters, and the functional ofscienti􀀌c interest admits a doubly robust estimating function. Doubly robust inference (Robins et al. , 1994) has received considerable interest in the past few years across multiple disciplines including statistics, epidemiology and econometrics. An estimator is said to be doubly robust if it remains consistent if one of two nuisance parameters needed for estimation is consistent, even if both are not necessarily consistent. The class of functionals that admit doubly robust estimators is quite rich, and includes estimation of pathwise di􀀋erentiable functionals in missing data problems under missing at random assumptions, and also in more complex settings where missingness pr

FAQs

What problem does the paper target in model selection?

It targets model selection in semiparametric inference when nuisance parameters can be high-dimensional, and standard selection based on prediction-oriented losses may not work for finite-dimensional functionals.

What is the core idea behind the proposed selection criteria?

Both criteria minimize a cross-validated quadratic pseudo-risk defined to embody double robustness, so the selected candidate model is biased-reduced relative to satisfying that property.

How does the paper justify the performance of the criteria?

It establishes an oracle property for the multi-fold cross-validation version, showing the empirical criteria perform nearly as well as an oracle that knows the pseudo-risk for each candidate model.

How is the approach used in practice?

The paper applies it to selecting a semiparametric estimator of the average treatment effect by using an ensemble of candidate machine learners to address confounding in observational studies.

Aug 3, 2026

Upload by Ophelia