Program Value-Added - A Feasible Method for Providing Evidence on the Effectiveness of Multiple Programs Implemented Simultaneously in Schools

Oct 15, 2026

Uploaded by Mabel

Assesses whether an adaptation of value-added analysis (VAA) can produce credible evidence about the relative effectiveness of interventions delivered at scale in a large school district. Uses two datasets: one linking interventions received by underperforming students and another covering interventions for students in schools funded through discretionary resources. Finds the first dataset identifies interventions differing from the average, while the second yields unexpected patterns. Concludes that program VAA can guide district choices under defined conditions, with a risk of misleading results if assumptions fail.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

AccessFree to download

Introduction

Rigor versus Feasibility Trade-off

Methods and Data

Dataset 1: Underperforming students

Dataset 2: Discretionary-funds schools

Results

Relative intervention effectiveness

Counterintuitive findings and implications

Conclusion

Program Value-Added: A Feasible Method for Providing Evidence on the Effectiveness of Multiple Programs Implemented Simultaneously in Schools Robert Shand 1 , Stephen M. Leach2, Fiona M. Hollands3, Florence Chang2, Yilin Pan4, Bo Yan2, Dena Dossett2, Samreen Nayyer-Qureshi5, Yixin Wang3, and Laura Head3 American Journal of Evaluation 2022, Vol. 43(4) 584-606 © The Author(s) 2022 Article reuse guidelines: [sagepub.com/journals-permissions](sagepub.com/journals-permissions)[ ](sagepub.com/journals-permissions)[DOI: 10.1177/10982140211071017](DOI: 10.1177/10982140211071017)[ ](DOI: 10.1177/10982140211071017)[journals.sagepub.com/home/aje](journals.sagepub.com/home/aje) Abstract We assessed whether an adaptation of value-added analysis (VAA) can provide evidence on the relative effectiveness of interventions implemented in a large school district. We analyzed two datasets, one documenting interventions received by underperforming students, and one documenting interventions received by students in schools beneﬁting from discretionary funds to invest in speciﬁc programs. Results from the former dataset identiﬁed several interventions that appear to be more or less effective than the average intervention. Results from the second dataset were counterintuitive. We conclude that, under speciﬁc conditions, program VAA can provide evidence to help guide district decision-makers to identify outlier interventions and inform decisions about scaling up or disinvesting in such interventions, with the caveat that if those conditions are not met, the results could be misleading. Keywords data analysis, impact evaluation, PreK–12 education, quasi-experiment Introduction Over the past few decades, federal education policy has increasingly emphasized the need for the scientiﬁc evaluation of educational programs (ESSA, 2015; NCLB, 2001). This push is at least partially 1 American University, Washington, DC, USA 2 Jefferson County Public Schools, Louisville, KY, USA 3 Teachers College, Columbia University, New York, NY, USA 4 World Bank, Washington, DC, USA 5 Columbia University, New York, NY, USA Corresponding Author: Robert Shand, American University, 4400 Massachusetts Ave NW, Washington, DC 20016, USA. Email: [rshand@american.edu](rshand@american.edu) motivated by the fact that costs of educating students have risen dramatically over the past 50 years while educational gains have not risen proportionally (Deming & Figlio, 2016) . The Every Student Succeeds Act (ESSA) explicitly deﬁnes what counts as evidence of effectiveness for educational programs, emphasizing both methodological rigor and also relevance to the local context. However, given the countless interventions in which American students participate each day, it is impractical to expect each one can be evaluated using the most rigorous research designs such as randomized controlled trials (RCTs) or quasi-experiments. These methods are resource-intensive, difﬁcult to implement, and yet may still be uninformative (Lortie-Forgues & Inglis, 2019) . Further, they represent a somewhat narrow deﬁnition of rigor that is well suited to answering a speciﬁc set of questions, but have other limitations and may not provide the types of evidence educational decision-makers need to inform their everyday decisions (Donaldson, Christie, & Mark, 2014) . Additionally, such studies can only count among the top two tiers of ESSA evidence if the study population and context match sufﬁciently with those of the local decision-maker (USED, 2016) . This requirement also presumes that education decision-makers are equipped to make such assessments although past research has demonstrated that evaluating research evidence requires capacity and infrastructure beyond that available in many education agencies (Honig & Coburn, 2008) . But school and district leaders often seek data on program delivery and on student outcomes to help guide annual budgeting and planning (Levenson, Baehr,

FAQs

What is the main goal of the study?

The study evaluates whether an adapted value-added analysis (“program VAA”) can provide evidence on the relative effectiveness of multiple interventions implemented simultaneously in a large school district.

What datasets are used for the analysis?

Two datasets are analyzed: one records interventions received by underperforming students, and the other records interventions received by students in schools benefiting from discretionary funds to invest in specific programs.

What do the results show about intervention effectiveness?

The first dataset identifies interventions that appear more or less effective than the average intervention. The second dataset produces results that are counterintuitive compared to expectations.

Under what conditions can program VAA support decision-making?

Program VAA can help district decision-makers identify outlier interventions and inform scaling up or disinvesting decisions only when specific conditions are met; otherwise, results may be misleading.

Oct 15, 2026

Uploaded by Mabel