More power to you - Using machine learning to augment human coding for more efficient inference in text-based randomized trials

Aug 5, 2026

For randomized trials using text as the outcome, assessing treatment impact depends on manually coding documents for constructs of interest, a step that is time-consuming and restricts measurement to a small set of dimensions. The presented inferential framework increases power under a fixed human-coding budget by leveraging unscored documents as supplementary information. It combines causal inference, survey sampling, and machine learning through four steps: sample and code documents, train a model from automated text features, predict all document scores, and adjust final effects using residual human-model discrepancies, supported by simulations and an education trial.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages42

Reading Time1 h 46 min

AccessFree to download

Introduction

Motivation for text-based randomized trials

Framework overview and goal

arXiv :2309 . 13666v2 [ stat .ME] 1 Aug 2024 More power to you: Using machine learning to augment human coding for more efficient inference in text-based randomized trials Reagan Mozer∗1 and Luke Miratrix2 1 Bentley University 2 Harvard University Abstract For randomized trials that use text as an outcome, traditional approaches for assessing treatment impact require that each document first be manually coded for constructs of interest by trained human raters. This process, the current standard, is both timeconsuming and limiting: even the largest human coding efforts are typically constrained to measure only a small set of dimensions across a subsample of available texts. In this work, we present an inferential framework that can be used to increase the power of an impact assessment, given a fixed human-coding budget, by taking advantage of any “untapped” observations – those documents not manually scored due to time or resource constraints – as a supplementary resource. Our approach, a methodological combination of causal inference, survey sampling methods, and machine learning, has four steps: (1) select and code a sample of documents; (2) build a machine learning model to predict the human-coded outcomes from a set of automatically extracted text features; (3) generate machine-predicted scores for all documents and use these scores to estimate treatment impacts; and (4) adjust the final impact estimates using the residual differences between human-coded and machine-predicted outcomes. This final step ensures any biases in the modeling procedure do not propagate to biases in final estimated effects. Through an extensive simulation study and an application toa recent field trial in education, we show that our proposed approach can be used to reduce the scope of a human-coding effort while maintaining nominal power to detect a significant treatment impact. Keywords: text analysis, automated scoring, randomized controlled trial, causal inference ∗ Both authors were supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305D220032. We would like to thank James Kim and the READS lab at the Harvard Graduate School of Education, who provided the data for the application described in Section 5. The authors also thank Finale Doshi-Velez, Kelly McConville, Tirthankar Dasgupta, and Nicole Pashley for conversations and advice about survey sampling methods relevant to this work. Participants at SREE 2023 and AEFP 2024 also provided excellent feedback, for which we are grateful. 1 Introduction Kim et al. (2021) recently conducted a large-scale randomized controlled trial to evaluate the Model of Reading Engagement (MORE), a content literacy intervention, on young students’domain knowledge in science and social studies as reflected by their performance on an argumentative writing assessment. The researchers collected thousands of student-generated essays, which were then hand-coded by trained research assistants as a preliminary step to assessing treatment impact. This process, the current standard, is both time-consuming and limiting: researchers typically do not have the resources to human-code all the documents they would wish. Furthermore, the result is a massive simplification of the data; written language is far more rich than what can feasibly be extracted by a human rater. This kind of effort is not uncommon: experimental research in education routinely relies on text collected from survey responses, written compositions, interviews, and other forms of discourse as a means to test psychological theories and to evaluate instructional practices. In this work, we show how to use modern machine learning tools to assist with human coding efforts, allowing more researchers to use text as an outcome. We believe achieving this end is important: while difficult to use, text is a critical outcome to consider. In K-12 settings, for example, students’ academic success is in part de

FAQs

What problem does the paper address in text-based randomized trials?

It addresses the time-consuming limitation of manually coding each document for constructs, which constrains which dimensions can be measured.

How does the proposed framework increase statistical power under a fixed coding budget?

It uses unscored, untapped documents as supplementary data by training a machine learning model to predict human-coded outcomes and then incorporating residual discrepancies into the final impact estimates.

What are the four main steps of the method?

Select and code a document sample, build a model predicting human-coded outcomes from automatically extracted text features, generate machine-predicted scores for all documents and estimate treatment impacts, and adjust estimates using residual differences between human and machine predictions.

Aug 5, 2026

Upload by Lucas Martin