A Double Machine Learning Approach to Combining Experimental and Observational Data

Aug 5, 2026

Upload by Patrick

Experimental and observational studies often lack validity due to assumptions that cannot be directly tested. The document proposes a double machine learning framework that combines both data sources to detect assumption violations and to estimate treatment effects with consistency. It evaluates violations of external validity and ignorability under milder conditions. When exactly one assumption fails, it yields semiparametrically efficient estimators. A no-free-lunch result underscores the need to correctly identify which assumption is violated, and three case studies demonstrate practical relevance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages39

Reading Time1 h 38 min

AccessFree to download

Introduction

Validity and ignorability in experiments and observational studies

Related work and double machine learning foundation

Proposed double machine learning framework

Testing assumption violations

Estimating treatment effects

Theoretical properties

Asymptotic normality and unbiasedness

Tests for unobserved confounding

Empirical applications

Project STAR case study

Coronary Artery Surgery Study case study

Additional real-world case study

arXiv :2307 .01449v1 [ stat .ME] 4 Jul 2023 A Double Machine Learning Approach to Combining Experimental and Observational Data Marco Morucci* 1 , Vittorio Orlandi*2 , Harsh Parikh*3 , Sudeepa Roy3 , Cynthia Rudin3 , Alexander Volfovsky2 *denotes joint ﬁrst authors (in the alphabetical order of their last names) 1 Center for Data Science, New York University 2 Department of Statistical Science, Duke University 3 Department of Computer Science, Duke University Abstract Experimental and observational studies often lack validity due to untestable assumptions. We propose a double machine learning approach to combine experimental and observational studies, allowing practitioners to test for assumption violations and estimate treatment effects consistently. Our framework tests for violations of external validity and ignorability under milder assumptions. When only one assumption is violated, we provide semiparametrically efﬁcient treatment effect estimators. However, our no-free-lunch theorem highlights the necessity of accurately identifying the violated assumption for consistent treatment effect estimation. We demonstrate the applicability of our approach in three real-world case studies, highlighting its relevance for practical settings. Keywords: Data Fusion, Generalizability, External Validity, Observational Study 1 Introduction Experiments and observational studies are both indispensable for estimating treatment effects and investigating causal questions of interest. Experiments, or randomized control trials (RCTs), guarantee unbiased estimation of sample average treatment effects via the randomization of treatment across the experimental population. However, they are typically small, expensive, and slow to design and implement. Another concern involves the experimental units themselves, which may be recruited via a non-randomized process and therefore may not be representative of the superpopulation of interest; in such cases we would say that the experiment lacks external validity. If this is the case, then the conclusions of the experiment may not be generalizable to the superpopulation, which is oftentimes of primary interest. On the other hand, observational studies, in which units self-select into treatment, tend to be larger and consist of units that better represent the superpopulation than experimental units. However, analysis of observational studies is plagued by observed and unobserved confounders – covariates that affect both units' propensity to self-select into treatment and their response. While a variety of methods – matching and weighting being two large classes among them – can be used to adjust for confounders, they typically rely on the crucial assumptions that they can accurately model the treatment-covariate or response-covariate relationships and that the available covariates are sufﬁcient for doing so. This latter assumption, which is often referred to as strong or conditional ignorability, is untestable using observational data alone [Rosenbaum and Rubin, 1983] . In this paper, we propose methods for jointly using experimental and observational data to (i) test for violations of at least one of validity and ignorability and (ii) estimate population average treatment effects even under such violations. The scope of our work is broader than that of previous work, which typically does not attempt the ﬁrst task and only focuses on a subset of the second, where external validity is violated but conditional ignorability isnot. Crucially, our estimators are also valid under violations of certain sets of assumptions that are often required in the literature, making our methods applicable in settings where other methods may be inappropriate. We provide a detailed discussion of the literature in Section 2 . Broadly, our methods build on the double machine learning literature [Chernozhukov et al., 2018] by estimating the nuisance parameters of interest (e.g., the propensity score) and

FAQs

Why do experiments and observational studies often face validity problems?

Experiments can suffer from external validity issues when sampled units are not representative of the target superpopulation. Observational studies are affected by confounding from both observed and unobserved factors that influence treatment selection and outcomes.

What does the proposed double machine learning approach achieve?

It jointly uses experimental and observational data to test for violations of external validity and ignorability and to estimate population average treatment effects even when such violations occur.

What is the key message of the no-free-lunch theorem in this framework?

Consistent treatment effect estimation requires correctly identifying which assumption is violated; otherwise the estimation can fail despite the use of advanced methods.

Aug 5, 2026

Upload by Patrick