Improved Power in a Reanalysis of an ASSISTments Efficacy Trial - Leveraging Auxiliary Data

Oct 15, 2026

Randomized controlled trials in education and biomedicine often discard nonrandomized auxiliary data because of confounding risk. This study re-analyzes an online homework RCT by combining randomized and nonrandomized subjects without introducing confounding. A random forest model trained on observational covariates predicts student test scores, and predicted outcomes are added to a hierarchical linear model. The approach reduces the standard error by 15%, comparable to increasing sample size by over one third, and adapts wherever auxiliary observational datasets exist.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Abstract

1. Introduction

Background: auxiliary data in RCTs and other trials

Prior work: auxiliary-data prediction and power gains

Gaps addressed by the current study

Improved Power in a Reanalysis of an ASSISTments Efficacy Trial by Leveraging Auxiliary Data Adam C. Sales Worcester Polytechnic Institute [asales@wpi.edu](asales@wpi.edu) Chun-Wei Huang WestEd [chuang@wested.org](chuang@wested.org) Mingyu Feng WestEd [mfeng@wested.org](mfeng@wested.org) Johann A. Gagnon-Bartsch University of Michigan [johanngb@umich.edu](johanngb@umich.edu) ABSTRACT Randomized controlled trials (RCTs) in social or biomedical science often rely on covariate and outcome data drawn from a larger administrative database. For instance, a recent paired, clustered RCT of an online homework platform used covariates (demographics and prior achievement measures) and outcomes (standardized test scores) from the state’s educational database. To avoid confounding, typical analyses use only data from randomized subjects, discarding the remainder of the database. In this study, we re-analyze data from the online homework study, substantially improving the precision of the effect estimates by using data from both randomized and nonrandomized subjects. Our estimator does not introduce confounding and relies largely on standard modeling techniques. Specifically, we use observational data to train a random forest model predicting student test scores from student- and school-level covariates, and use it to generate predicted outcomes for students in the randomized schools. Then, we modify the hierarchical linear model from the original analysis by adding the predicted outcomes as an additional regressor, reducing the standard error by 15%—the improvement expected from increasing the sample size by over a third. Since it requires few additional modeling skills and no additional assumptions, this method can be easily adapted wherever auxiliary observational data are available. Keywords Causal Inference, Randomized Experiments, Supervised Learning, Hierarchical Linear Models 1. INTRODUCTION In 2018, WestEd, an educational research agency, began a large-scale randomized field trial of the online mathematics application ASSISTments in 63 schools in North CarAdam Sales, Kevin Huang, Mingyu Feng, and Johann GagnonBartsch. Improved Power in a Reanalysis of an ASSISTments Efficacy Trial by Leveraging Auxiliary Data. In Anthony Botelho, Maria Mercedes T. Rodrigo, Adish Singla, Hiroaki Ogata, Hyojeong So, and Young Hoan Cho (eds.) Proceedings of the 19th International Conference on Educational Data Mining, Seoul, Republic of Korea, June, 2026, pp. 487–493. International Educational Data Mining Society (2026) . © 2026 Copyright is held by the author(s) . This work is distributed under the Creative Commons Attribution NonCommercial NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license. [https://doi.org/10.5281/zenodo.21039716](https://doi.org/10.5281/zenodo.21039716) olina [3] . The researchers were granted access to the North Carolina student longitudinal data system, from which they drew most of the data used for the evaluation, including demographic and student achievement data from the students in the study—seventh-grade math students in the 63 randomized schools. However, the researchers had access to even more data: demographic, prior achievement, and outcome data from over 65 thousand additional 7th-grade students who attended North Carolina middle schools other than those 63 . The WestEd ASSISTments study is one example of a common phenomenon: covariate and outcome data for a randomized controlled trial (RCT) are drawn from a larger database that also includes data from similar subjects who were not randomized as part of the experiment. Other examples include A/B tests, which draw baseline covariatesand outcomes from log data, and clinical trials, which draw baseline and outcome data from patients’ medical records. In these examples, researchers have access to big auxiliary observational datasets that are typically discarded or left unused. Granted, pooling data from randomized and nonran

FAQs

Why do standard analyses of randomized trials usually discard nonrandomized auxiliary data?

Because auxiliary data can contain subjects that are not guaranteed to be comparable to the randomized participants, creating confounding and undermining the RCT’s purpose.

How does the proposed reanalysis incorporate auxiliary data without introducing confounding?

It trains a random forest on observational covariates to predict test scores, then adds these predicted outcomes as an additional regressor in a hierarchical linear model.

What power or precision improvement does the method achieve?

The standard error is reduced by 15%, which corresponds to the improvement expected from increasing the sample size by more than one third.

Oct 15, 2026

Uploaded by Aria Callaghan