High Stakes, but Low Validity? A Case Study of Standardized Tests and Admissions into New York City Specialized High Schools

Sep 15, 2026

Uploaded by Jake

A policy brief examines how admissions to select New York City public specialized high schools rely solely on the SHSAT, as required by New York State law. Using individual test results from 2005 and 2006, it evaluates scoring behavior, uncertainty near cutoff scores, version-to-version differences, and gaps in research on bias. Findings argue the current system may disadvantage students with limited access to test preparation while lacking evidence of validity or clear SHSAT objectives.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages37

Reading Time1 h 33 min

AccessFree to download

Executive Summary

Admissions process and legal requirement

Findings on scoring, validity, and statistical uncertainty

Concerns about test versions and potential bias

Psychometric standards and predictive validity rationale

| High Stakes, but Low Validity? A Case Study of Standardized Tests and Admissions into New York City Specialized High Schools
Joshua Feinman, Ph.D.
October 2008
EPRU | EDUCATION POLICY RESEARCH UNIT
| | | --- | --- | | Education Policy Research Unit
Division of Educational Leadership and Policy Studies College of Education, Arizona State University
P.O. Box 872411, Tempe, AZ 85287-2411 Telephone: (480) 965-1886
Fax: (480) 965-0303
E-mail: [epsl@asu.edu](epsl@asu.edu)
[http://edpolicylab.org](http://edpolicylab.org) | Education and the Public Interest Center
School of Education, University of Colorado Boulder, CO 80309-0249 Telephone: (303) 447-EPIC
Fax: (303) 492-7090
Email: [epic@colorado.edu](epic@colorado.edu)[ ](epic@colorado.edu)[http://epicpolicy.org](http://epicpolicy.org) | | ● Suggested Citation:
Feinman, J. (2008). High Stakes, but Low Validity? A Case Study of Standardized Tests and Admissions into New York City Specialized High Schools. Boulder and Tempe: Education and the Public Interest Center & Education Policy Research Unit. Retrieved [date] from
[http://epicpolicy.org/publication/high-stakes-but-low-validity](http://epicpolicy.org/publication/high-stakes-but-low-validity) | | EPIC/EPRU policy briefs are peer reviewed by members of the Editorial Review Board. For information on the board and its members, visit: [http://epicpolicy.org/editorial-board](http://epicpolicy.org/editorial-board) High Stakes, but Low Validity? A Case Study of Standardized Tests and Admissions into New York City Specialized High Schools 1 Joshua Feinman Executive Summary This is a study of the admissions process at a select group of New York City public high schools. It offers the first detailed look at the admissions practices of this highly regarded and competitive group of schools, and also provides a window into the broader national debate about the use of standardized tests in school admissions. According to New York State law, admission to these schools must be based solely on an exam. The exam used is called the Specialized High Schools Admissions Test (SHSAT) . This study makes use of the individual test results from 2005 and 2006. Several key findings emerge: • The SHSAT has an unusual scoring feature that is not widely known, and may give an edge to those who have access to expensive test-prep tutors. Other reasonable scoring systems could be constructed that would yield different results for many students, and there is no evidence offered to support the validity of the current system. • Thousands of students who are not being accepted have scores that are statistically indistinguishable from thousands who are granted admission. And these estimates are derived using the less precise, classical-test-theory-based measures of statistical uncertainty, which may understate the problem. The New York City Department of Education (NYCDOE) fails to provide the more accurate, itemresponse-theory-based estimates of the SHSAT’s standard error of measurement (SEM) near the admission cutoff scores, which would offer a clearer picture of how well the test is able to differentiate among students who score close to the admission/rejection line. This omission violates generally-accepted testing standards and practices. • Students who receive certain versions of the test may be more likely to gain admission than students who receive other versions. No evidence is offered on how accurate the statistical equating of different test versions is. The mean scaled scores vary across versions much more than would be expected given the chance distribution of ability across large random samples of students, suggesting that the scoring system may not be completely eliminating differences among test versions. • No studies have ever been done to see if the SHSAT is subject to prediction bias across gender and ethnic groups (i.e., if SHSAT scores predict things for different groups) . Of course

FAQs

What admissions rule governs entry to New York City specialized high schools?

Admission must be based solely on an exam, the Specialized High Schools Admissions Test (SHSAT), under New York State law.

What concerns does the study raise about the SHSAT scoring system?

It highlights an unusual scoring feature that is not widely known, notes a lack of evidence supporting the current system’s validity, and suggests alternative scoring could produce different outcomes.

What research gaps does the study identify regarding validity and bias?

It reports no predictive validity studies by the NYCDOE to show how the test performs and no studies examining prediction bias across gender and ethnic groups.

Sep 15, 2026

Uploaded by Jake