Educational Evaluation and Policy Analysis - Learning From Cluster Randomized Trials in Education: An Assessment of the Capacity of Studies to Determine What Works, For Whom, and Under What Conditions

Oct 15, 2026

Uploaded by dabgoudw

Educational Evaluation and Policy Analysis presents an assessment of how well cluster randomized trials (CRTs) can deliver rigorous evidence about effectiveness. The study analyzes the capacity of CRTs to answer what works, for whom, and under what conditions, considering how design choices influence detectability of treatment moderators. Findings indicate stronger likelihood of detecting meaningful individual-level moderator effects, such as gender, compared with cluster-level moderators, such as school type.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Introduction

What Works, For Whom, Under What Conditions?

Cluster Randomized Trials and Moderator Effects

Design Capacity for Evidence

Changes in IES Requests for Applications

Statistical Power Requirements

Educational Evaluation and Policy Analysis September 2020, Vol. 42, No. 3, pp. 354–374 DOI: 10.3102/0162373720929018 Article reuse guidelines: [sagepub.com/journals-permissions](sagepub.com/journals-permissions)[ ](sagepub.com/journals-permissions)© 2020 AERA. [http://eepa.aera.net](http://eepa.aera.net) Learning From Cluster Randomized Trials in Education: An Assessment of the Capacity of Studies to Determine What Works, For Whom, and Under What Conditions Jessaca Spybrook Qi Zhang Western Michigan University Ben Kelcey University of Cincinnati Nianbo Dong The University of North Carolina at Chapel Hill Over the past 15 years, we have seen an increase in the use of cluster randomized trials (CRTs) to test the efficacy of educational interventions. These studies are often designed with the goal of determining whether a program works, or answering the what works question. Recently, the goals of these studies expanded to include for whom and under what conditions an intervention is effective. In this study, we examine the capacity of a set of CRTs to provide rigorous evidence about for whom and under what conditions an intervention is effective. The findings suggest that studies are more likely to be designed with the capacity to detect potentially meaningful individual-level moderator effects, for example, gender, than cluster-level moderator effects, for example, school type. Keywords: educational policy, program evaluation, evaluation, experimental design, research methodology In the past 15 years, we have seen an increased emphasis on the use of randomized trials (RTs), particularly cluster randomized trials (CRTs), to test the efficacy of educational interventions (Spybrook et al., 2016) . In a CRT, entire clusters, most often schools, are randomly assigned to condition. The most prominent funder of CRTs to assess the efficacy of educational interventions is the Institute of Education Sciences (IES), the research branch of the U.S. Department of Education. Since its inception in 2002, IES has played a leadership role in shaping education policy and practice around the use of RTs and CRTsto assess the efficacy of educational programs (Cook & Foray, 2007) . As the leading funder of education research, IES has funded more than 250 efficacy studies, many of which are CRTs. Although there certainly are other federal funders of CRTs aside from IES, for example, the National Science Foundation (NSF) and the Office of Investment and Innovation (OII), IES has undoubtedly played a leadership role in the movement toward the use of CRTs to test educational interventions and as such is featured prominently in this article. IES was established by the Education Science Reform Act of 2002. The mission of IES is to build a body of rigorous evidence to inform education policy and practice ([http://ies.ed.gov](http://ies.ed.gov)). In the early years, this meant that IES focused on What Works, For Whom, Under What Conditions? answering the what works question. For example, what math curriculum is most effective for improving math achievement? What reading professional development program is most effective for improving reading achievement? IES prioritized efficacy studies that included RTs, most often CRTs, to answer this question. Although answering the what works question is still a primary goal of efficacy trials, over time we have seen the goals expand to questions about for whom and under what conditions. These types of questions are critical in that they help provide amore comprehensive picture about the types of students and schools that the programs are most impactful. For example, a new math program may be more effective for girls than boys or small schools rather than large schools. This is important information for school administrators as they consider whether or not an intervention will work in their schools and for their students. These questions are also central to the mission of IES to improve ou

FAQs

What question does the article focus on regarding education interventions?

It examines the capacity of cluster randomized trials to provide rigorous evidence about what works, for whom, and under what conditions interventions are effective.

How do the findings compare individual-level versus cluster-level moderator effects?

The studies are more likely to be designed with the capacity to detect potentially meaningful individual-level moderator effects (e.g., gender) than cluster-level moderator effects (e.g., school type).

Which methodological component does the article highlight when discussing changes in expectations?

It focuses on statistical power to detect effects, specifically main effects and moderator effects, using IES requests for applications from 2002 to 2017.

Oct 15, 2026

Uploaded by dabgoudw