(In)Stability of Test Scores - Paper Findings - Simple Bounce Statistic for Test Score Stability

Oct 15, 2026

Both school and district administrators rely on standardized large-scale test results to decide whether educational programs and interventions are needed or successful. Yet school-level outcomes fluctuate from year to year due to changing student cohorts, test items, and other uncontrollable factors. The study analyzes Ontario, Canada school-level results and uses G-theory to show scores are insufficiently stable for defensible conclusions. It recommends collecting more data for small and medium schools and introduces a bounce statistic for easier stability interpretation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Abstract

Introduction

Purpose of standardized testing in Ontario

Uses and misuses of EQAO results

Consequences for school improvement and leadership

Need for data literacy and limits of interpretation

Method overview and findings (from abstract)

(In)Stability of Test Scores Stefan Merchant1, Jessica Rich1, & Don A. Klinger2 1Queen’s University, 2 University of Waikato Abstract Both school and district administrators use the results of standardized, large-scale tests to inform decisions about the need for, or success of, educational programs and interventions. However, test results atthe school level are subject to random fluctuations due to changes in cohort, test items, and other factors outside of the school’s control. This study examined year to year changes in school level results on standardized tests delivered in Ontario, Canada. G-theory analyses found that test scores are not stable enough for meaningful conclusions to be made based on year to year changes in school level results. For small and medium sized schools, years of data need to be collected before defensible decisions can be made about trends in test scores. The authors introduce a ‘bounce’ statistic that provides a simple, easy to interpret measure of test score stability. Keywords: large-scale testing, G-theory, educational policy, test reliability Introduction All provinces have testing programs to help them evaluate student achievement in relation to curriculum expectations (e.g., British Columbia, 2021; Prince Edward Island, 2019). These tests may be implemented in early years (e.g., Saskatchewan, 2021), in middle school (e.g., Manitoba, n.d.), or for graduation (e.g., Alberta, 2021). In Ontario, Grade 3 and 6 students are required to write province-wide literacy and numeracy tests. These tests originated from the Education Quality and Accountability Office (EQAO), which implement a variety of standardized tests in Ontario schools. While some of those tests are high stakes for students (e.g., Ontario Secondary School Literacy Test), the grade 3 and 6 tests are low stakes, as there are no sanctions, promotions, or report card grades associated with student performance. The primary purpose of the EQAO tests is to monitor overall student achievement against the provincial standards for reading, writing, and mathematics (Educational Quality and Accountability Office, 2020), but the intended and unintended purposes for the tests have expanded. As examples of intended uses, the Ontario Ministry of Education has directed funds and resources to schools with very low student performance on these tests. Test results have also been used to implement educational initiatives and policies within the province and to monitor the subsequent impact of these initiatives (Bolden et al., 2014; Canadian Language and Literacy Research Network, 2008; Klinger & Rogers, 2011; Klinger & Wade-Woolley, 2009). At the same time, unintended uses include examples of organizations that use the EQAO results to rank schools, to infer differences in educational quality between schools, or to infer which neighbourhoods have the best schools (e.g., Cowley & Emes, 2020; Scholarhood, 2017; Volante, 2004). One Ontario realty website ([http://www.realtyforsale.ca/best_schools/](http://www.realtyforsale.ca/best_schools/)) even goes so far as to list local elementary schools using a traffic light system of red (for low ranking schools), yellow, and green (for high ranking schools). Ontario schools often use EQAO test results to establish, monitor, and evaluate school improvement plans. Hence it is common for schools to report increases in their EQAO test results as evidence of the impact of improvement plans and the lack of improvement as the need for renewed efforts (e.g., Calder, 2015; Renfrew County District School Board, 2016; Toronto District School Board, 2018). Such uses can result in high stakes for teachers and administrators. Elementary school principals have anecdotally told us they are evaluated based upon EQAO test results; consequently, they make leadership decisions designed to augment the scores. These decisions include which teachers get placed into Grades 3 and 6 (grades in which the EQAO tests are adminis

FAQs

Why do administrators use standardized test results for education decisions?

They use the results to determine the need for or success of educational programs and interventions, and to monitor achievement against provincial standards.

What causes year-to-year instability in school-level test outcomes?

Random fluctuations arise from changes in student cohorts, test items, and other factors outside schools’ control.

What do the G-theory analyses show about using year-to-year school results?

They indicate that test scores are not stable enough to support meaningful, defensible conclusions based on year-to-year school-level changes.

Oct 15, 2026

Uploaded by Danger Angel