Stakes and Signals - An Empirical Investigation of Muddled Information in Standardized Testing - Working Paper No. 32608

Sep 5, 2026

Upload by Tawan

Examines a natural experiment in Brazil where similar students took the same standardized test either as a low-stakes school accountability exam or as a high-stakes admission exam for top universities. Using administrative data and a difference-in-differences design, the study finds that score gaps between high- and low-income students widen under higher stakes, consistent with increased test preparation by wealthier students. The higher-stakes setting also makes scores more informative about college outcomes. Overall, the “muddling” between ability and prep improves the quality of the score signal while exacerbating inequality.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages46

Reading Time1 h 56 min

AccessFree to download

Introduction

NBER WORKING PAPER SERIES STAKES AND SIGNALS: AN EMPIRICAL INVESTIGATION OF MUDDLED INFORMATION IN STANDARDIZED TESTING Germán J. Reyes Evan Riehl Ruqing Xu Working Paper 32608 [http://www.nber.org/papers/w32608](http://www.nber.org/papers/w32608) NATIONAL BUREAU OF ECONOMIC RESEARCH 1050 Massachusetts Avenue Cambridge, MA 02138 June 2024 For helpful comments we thank Serena Canaan, William Dougan, Deborah Doukas, Jorge Luis García, Daniel Hamermesh, Navin Kartik, Pierre Mouganie, Eric Nielsen, Roberta Olivieri, Christiane Szerman, Christopher Walters, Russell Weinstein, and participants at various seminars and conferences. Adya Bhargava provided excellent research assistance. All errors are our own. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research. NBER working papers are circulated for discussion and comment purposes. They have not been peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies official NBER publications. © 2024 by Germán J. Reyes, Evan Riehl, and Ruqing Xu. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including © notice, is given to the source. Stakes and Signals: An Empirical Investigation of Muddled Information in Standardized Testing Germán J. Reyes, Evan Riehl, and Ruqing Xu NBER Working Paper No. 32608 June 2024 JEL No. I23,I24,J24,M5 ABSTRACT We examine a natural experiment in Brazil in which similar students took the same standardized test as either a low-stakes school accountability exam or a high-stakes admission exam for the country's top universities. Using administrative data and a difference-in-differences design, we find that test score gaps between high- and low-income students expanded on the high-stakes exam, consistent with wealthy students engaging in test prep. Yet the increase in stakes made scores more informative for students' college outcomes. Thus the "muddling" of information on natural ability and test prep improved the quality of the score signal, although it also exacerbated inequality. Germán J. Reyes Middlebury College 1229 S Street Extension Middlebury, VT 05753 [greyes@middlebury.edu](greyes@middlebury.edu) Evan Riehl Department of Economics Cornell University 266 Ives Hall Ithaca, NY 14853 and NBER [eriehl@cornell.edu](eriehl@cornell.edu) Ruqing Xu Cornell University [rx24@cornell.edu](rx24@cornell.edu) A data appendix is available at [http://www.nber.org/data-appendix/w32608](http://www.nber.org/data-appendix/w32608) 1. Introduction Selecting the right talent is crucial to the success of organizations. A fundamental challenge faced by recruiters is incomplete information on candidate quality. To address this challenge, recruiters often rely on signals of candidate quality from mechanisms like standardized tests or case interviews. Yet candidates have an incentive to manipulate their signals through preparation or even outright cheating, particularly when the positions they are applying for are highly desirable.1 An important question for the design of talent selection mechanisms is whether the potential for manipulation degrades the quality of the signal. Social scientists have long hypothesized that strategic manipulation erodes the reliability of signals, an idea that is known as Goodhart’s Law (Goodhart, 1975) or Campbell’s Law (Campbell, 1979) .2 Theoretical models of “muddled information”(Frankel and Kartik, 2019) show that, as the stakes of a mechanism increase, signals become more informative about an individual’s gaming ability and less informative about the natural action individuals would take absent signaling concerns.3 But it is theoretically ambiguous whether recruiters would prefer to observe signals of candidate quality with or without gaming. Gaming ability may reﬂect a candidate’s know

FAQs

What natural experiment does the paper analyze in Brazil?

The paper studies Brazil where a test originally used as a low-stakes school accountability measure was repurposed into a high-stakes university admission exam, while keeping the test structure and taker composition fixed.

How do higher exam stakes affect gaps between high- and low-income students?

Higher stakes expand test score gaps between high- and low-income students, consistent with wealthier students engaging more in test preparation.

Do stakes make test scores more or less informative for college outcomes?

Higher stakes make test scores more informative for students’ college outcomes, even though they worsen inequality.

Sep 5, 2026

Upload by Tawan