IRT MODELS - Maintaining Standards Over Time - Introduction

Oct 20, 2026

The text analyzes how educational authorities set and maintain grading standards over time, focusing on the UK GCE/GCSE context and contrasting it with US practices. It explains the “contest model” as a judgment-and-statistics approach, then critiques purely statistical methods for lacking inspection of work and relying on subjective variable choices. It argues that once standards are set, IRT-based test equating can help maintain them by using candidate performance on subsets of items, reducing dependence on bias-prone predictions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Introduction

Maintaining Standards Over Time

IRT Models

INTRODUCTION At the end of the GCE Accounting awarding meeting in July 2007, the Chair of Examiners for Accounting reported that while the committee had followed the statistical recommendations on the standards that had been put to them to consider, they “would have welcomed the opportunity to revise standards in the candidates’ favour”. Chair of Examiners Report, GCE Accounting, June 2007 Unlike the GCE English committee in 1990 who decided to cut the pass rate at Grade A from 5.7% to 0.7% in one fell swoop (Baird, 2007) this committee were not given that opportunity. Their task was twofold. They were required to use their judgement to examine specific candidates’ performance on the test content to determine the extent to which they may have been affected by any changes in the difficulty of an examination from the previous year; but in making any compensation for that perceived change, they were to take into account the statistical information which informed them about various characteristics of the cohort taking that examination. This model of setting and maintaining standards has been dubbed rather derogatorily, but perhaps aptly, the contest model (Newton, 2005), as it requires participants to bring together sources of evidence that occasionally conflict. This process, however, is the essence of any standard setting procedure: it requires participants to bring to bear information about both test content and test takers using a combination of judgement and statistics. What is unique to the model used in the UK is that a single methodology has developed for setting standards and maintaining standards. In this US there are two very different traditions for standard setting (see Cizek & Bunch, 2007 for an review of methodologies) and maintaining standards, which is largely done using test equating (see Kolen & Brennan, 2004 for an overview of methodologies) . The Accounting committee, having set the initial height of the bar in 2002, have been given the opportunity to spend five years wrangling over whether it was indeed set at the right height, when that height may not even exist in any objective sense! This paper argues that once standards have been set, IRT methods of test equating could be used to maintain them. MAINTAINING STANDARDS OVER TIME If it is accepted that standards do not exist in any objective sense as they represent the expectations of society and reflect the uses to which those standards may be put (Baird, Cresswell, & Newton, 2000) then no purely statistical technique could ever be substituted for an initial standard setting process. Once a standard has been set, however, the task of maintaining that standard over short periods of time should be fairly straightforward. In practice the weight of evidence currently rests firmly with statistical methods (Stringer, 2008a) . These methods assume that last year’s outcome provides a statistical starting point for standard setting this year, ceteris Paribas (Cresswell, 1992) . There are two fundamental problems with a purely statistical approach, however. Firstly, no examination work is ever inspected. The system could go ahead blithely awarding grades based on predictions without candidates ever turning up for examinations. Secondly, the choice of variables that compose any statistical method of maintaining pass rates is a subjective decision. Under the present system, for example, no statistical controls are made for different rates of progress by gender. As a result over five hundred committees of eight senior examiners (on average) are convened to make judgements of students’ performance on GCSE and GCE examinations in England every year (Baird & Dhillon, 2005) . Work is inspected, and the statistically predicted boundaries ratified in almost all cases (Stringer, 2008a) . Occasionally the judgment suggests that alternative statistical indictors be sought, but this useful purpose has to be set against the evidence that giving the candidates the be

FAQs

What is the main idea behind the UK standard-setting approach described in the text?

The approach combines judgment and statistics so that participants reconcile potentially conflicting sources of evidence about test content and examinees.

Why does the text argue that purely statistical maintenance is problematic?

It notes that no examination work is inspected and that statistical models depend on subjective choices of variables, including inadequate controls for factors such as gender-related progress rates.

How does the text propose maintaining standards after initial standard setting?

It argues that IRT methods of test equating can be used to maintain standards by leveraging how candidates perform on subsets of test items rather than relying mainly on prediction-based boundaries.

Oct 20, 2026

Uploaded by Quinn Holloway