Metric-Free Measures of Test Score Trends and Gaps - CSE Report 665 - with Policy-Relevant Examples

Sep 15, 2026

The report addresses policy decisions that rely on comparing test score changes over time and assessing how group differences and gap changes evolve. It proposes and develops metric-free methods that remain invariant under positive monotonic transformations, enabling meaningful comparisons when test score scales lack defensible interval properties. Across a three-part framework, it demonstrates advantages of metric-free analysis and applies it to persistent achievement gaps and trends between California STAR and NAEP reading outcomes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages54

Reading Time2 h 16 min

AccessFree to download

Part I: A Metric-Free Framework

Distinguishing change, gaps, and changes in gaps

Metric-free comparisons across different score scales

Metric-Free Measures of Test Score Trends and Gaps with Policy-Relevant Examples CSE Report 665 Andrew D. Ho and Edward H. Haertel CRESST/Stanford University January 2006 Center for the Study of Evaluation National Center for Research on Evaluation, Standards, and Student Testing Graduate School of Education & Information Studies University Of California, Los Angeles Los Angeles, CA 90095-1522 (310) 206-1532 Copyright © 2006 The Regents of the University of California Project 3.5 Methodological Issues in Accountability Systems, Strand 1: The Behavior of Linking Items in Test Equating (1st 2 years) . Project Directors: Andrew D. Ho and Edward H. Haertel, CRESST/Stanford University The work reported herein was supported in part under the Educational Research and Development Centers Program, PR/Award Number R305B960002, as administered by the Institute of Education Sciences, U.S. Department of Education. The findings and opinions expressed do not reflect the positions or policies of the National Center for Education Research, the Institute of Education Sciences or the U.S. Department of Education. . METRIC-FREE MEASURES OF TEST SCORE TRENDS AND GAPS WITH POLICY-RELEVANT EXAMPLES Andrew D. Ho and Edward H. Haertel CRESST/Stanford University Problems of scale typically arise when comparing test score trends, gaps, and gap trends across different tests. To overcome some of these difficulties, we can express the difference between the observed test performance of two groups with graphs or statistics that are metric-free (i.e., invariant under positive monotonic transformations of the test score scale) . In a series of studies broken into three parts, we develop a framework for the application of metric-free methods to routine policy questions. The first part introduces metric-free methodology and demonstrates the advantages of these methods when test score scales do not have defensible interval properties. The second part uses metric-free methods to compare gaps in HispanicWhite achievement in California across four testing programs over a 7-year period. The third part uses metric-free methods to compare trends for “high-stakes” State Reading test scores to State score trends on the National Assessment of Educational Progress from 2002 to 2003. As a whole, this series of studies represents an argument for the usefulness of metric-free methods for quantifying trends and gaps and the superiority of metric-free methods for comparing trends and gaps across tests with different score scales. Part I: A Metric-Free Framework As federal educational policies are implemented by states, two questions are being placed as cornerstones of educational decision-making: a) How are test scores changing over time? and b) How are differences between the test scores of certain groups changing over time? To use straightforward terminology, we distinguish here among “change”(Question 1), and “gaps,” and “changes in gaps”(Question 2) . In smaller, policy-research circles, a third, higher order question arises: Are answers to the first two questions the same for different tests? Studies that address this third question (Klein, Hamilton, McCaffrey & Stecher, 1998; Koretz & Barron, 1998; Linn, Graue, & Sanders, 1990; and many more) have received increased attention as the demand for external validation of high-stakes test score gains has increased. These questions might be addressed in terms of averages. We can discuss the policy-relevant averages with a running example. Call X a Grade 4 Reading Test score from California’s Standardized Testing and Reporting (STAR) system, and call Y a California Grade 4 Reading Test score on the National Assessment of Educational Progress (NAEP)1. Subscripts a and b correspond to scores from individuals from an advantaged and a disadvantaged group respectively, and subscripts 1 and 2 correspond to the 2002 and 2003 administrations of the tests respectively. Score differences are not

FAQs

How are the STAR and NAEP tests used in the framework?

The report uses STAR scores (California Grade 4 Reading) and NAEP scores for 2002 and 2003 to build comparisons, including a discrepancy concept that contrasts changes on STAR with changes on NAEP.

Sep 15, 2026

Uploaded by Aria Callaghan