Using Adaptive Comparative Judgement to Obtain a Highly Reliable Rank Order in Summative Assessment

Oct 20, 2026

Adaptive Comparative Judgment (ACJ) provides an alternative to traditional marking, particularly for performance assessment where achievements are hard to codify in mark schemes. ACJ delivers pairs of candidates’ work via a web browser for expert judges to compare against educational objectives. Judges’ selections are analysed with the Rasch logistic model to estimate a quality parameter for each work, enabling placement on a measurement scale. An adaptive process selects comparisons to maximize information and support reliable rank ordering.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Adaptive Comparative Judgement

Comparative judgement

Centre for Education Research and Policy Using Adaptive Comparative Judgement to Obtain a Highly Reliable Rank Order in Summative Assessment Claire Whitehouse and Alastair Pollitt Abstract Adaptive Comparative Judgment (ACJ) offers an alternative to marking, especially for performance assessments for which achievement can be difficult to describe in mark schemes. The ACJ system uses a web-browser to deliver pairs of candidates’ work to judges. The judges apply their subject expertise to select which of the two sets of work is better in terms of meeting appropriate educational objectives. A quality parameter is estimated for each piece of candidates’ work using the Rasch logistic model to analyse the judges’ decisions. Placing the pieces of work on a measurement scale gives them a rank order. The adaptive nature of ACJ lies in the pairing of essays for each judge. The selection of essays is based on a maximum distance between the quality parameter estimates, allowing useful information to be gained from each paired comparison. This paper reports a study in which ACJ was used to rank order a random sample of 564 essays on a topic in physical geography based on the judgements of a group of 23 teachers and examiners. The reliability of the rank order was 0.97. Evidence is presented for the need for judges to be teaching at the same qualification level as they are judging at. There is a discussion of the factors that need to be addressed before implementing ACJ in summative assessment. Keywords: adaptive comparative judgement, Rasch model, summative assessment Introduction Comparative judgement Adaptive Comparative Judgement (ACJ) offers an alternative to marking. It originates in the method of comparative judgement (CJ) that was first proposed by Thurstone (1927) as a description of the processes of human judgement that are not visible to the observer. In stating the model of comparative judgement, Thurstone suggested that it could be used to quantify the perceived quality of objects, including handwriting specimens and children’s drawings. Thurstone’s model has five versions. The simplest and most tractable is based on three assumptions. The first assumption is psychological in nature: that the judgements being made are holistic and above all comparative. As Laming (2011) states “ There is no absolute judgement. All judgements are comparisons of one thing with another.” When a judge is asked to make an absolute judgment about the perceived quality of an object, previous experience, level of knowledge, self-efficacy and the opinions of others all influence that judgement. In summative assessment, examiners are (and should be) greatly influenced by the mark scheme to an extent that overcomes bias as far as possible. So the absolute judgement of what mark to award is relative to the mark scheme plus any error and bias in its interpretation. A series of absolute judgements results in a number of measurements on the scale of perceived quality, and when they are made by a group of judges the leniency or severity with which each [www.cerp.org.uk](www.cerp.org.uk) Copyright © 2012 AQA and its licensors. All rights reserved. The Assessment and Qualifications Alliance (AQA) is a company limited by guarantee registered in England and Wales (company number 3644723) . Registered address: AQA, Devas Street, Manchester M15 6EX judge makes their judgements will be visible. This bias is one of the threats to the validity and reliability of marking. In comparing two objects and deciding which one is the ‘better’ of the two, however, it does not matter where a judge would locate the quality of the individual objects on a scale of quality. What matters is the difference in quality between the two objects. So, in placing students’work on a measurement scale the bias of individual judges is unimportant so long as they are able to perceive differences in quality based on a set of shared criteria. The cancelling out of the

FAQs

What does Adaptive Comparative Judgement (ACJ) replace in summative assessment?

ACJ offers an alternative to marking. It is designed to assess performance when describing achievement in mark schemes is difficult.

How does ACJ generate a rank order from judges’ comparisons?

ACJ presents judges with pairs of candidates’ work and uses their comparative decisions. A quality parameter for each piece is estimated using the Rasch logistic model, then placements on a measurement scale produce a rank order.

What was the reliability of the ACJ-derived rank order in the study?

The reliability of the rank order reported for the sample of 564 essays was 0.97.

Oct 20, 2026

Uploaded by Jordan Avery