Levels-based mark schemes and marking bias - Reliability and validity in assessment evaluation

Oct 20, 2026

Reliability and validity remain central to evaluating assessments. Drawing on research into question papers, mark schemes and examiner effects, the paper examines how levels-based mark scheme rules may introduce systematic error and affect unbiased measurement. It takes a statistical perspective on mark scheme design, arguing that the distribution of marks should be shaped by candidates’ responses to items. Pending further research, it recommends designing levels-based mark schemes so each band has equal width.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Summary

Keywords

Introduction

Classical test theory and measurement error

Prior research on marking and mark scheme design

Valid assessment and examiner-mark scheme interaction

Centre for Education Research and Policy Levels-based mark schemes and marking bias Anne Pinot de Moira Summary Reliability and validity are ever-present themes in the evaluation of assessment. Previous research studies have considered the role of the question paper, the mark scheme and the examiners in providing unbiased estimates of students’ ability. With particular reference to the mark scheme, this paper focuses on how the rules imposed by levels-based mark schemes might affect the extent to which a measurement can be considered unbiased. It considers neither cognitive strategies nor the role of judgement instead, from a purely statistical point of view, it investigates the potential impact of mark scheme design. It concludes that the responses to a question itself should shape the mark distribution, rather than the shape being determined by the structure of the mark scheme. To this end, and pending further research in the area, it recommends that levels-based mark schemes should be designed such that each band is of equal width. Keywords: Mark schemes, levels-based, banded, marking bias Introduction According to classical test theory the observed mark for a candidate is the sum of two independent parts, the true mark for that candidate and measurement error (Lord & Novick, 1968) . Observed Mark = True Mark + Measurement Error The theory assumes that the error is random and normally distributed. Thus, across a set of candidates, the distribution of observed marks should replicate that of the true marks, assuming that the errors are uncorrelated. A well-designed test should, therefore, provide a set of marks which reflect the true distribution of ability in a given discipline. However, effective measurement relies on effective test development and a test is only as good as its component parts. The reliability and validity of a test can be compromised at any stage of the process. For example, a poorly specified syllabus might affect the validity of the test and, likewise, a badly designed test might not measure what it purports to measure. Poor examiners or ambiguity in the test might preclude reliable marking and affect the extent to which the measurement error can be described as random but, so too, might an ineffective mark scheme. It is the influence of the mark scheme on effective measurement which provides the focus for this paper. There is a considerable bank of research considering marking, mark scheme design and the implications for reliable marking. Mark remark reliability studies have often been used as a tool to identify features of a mark scheme which might give rise to problems in marking (see for example Baird, Greatorex, & Bell, 2003; Black, Suto, & Bramley, 2011; Massey & Raikes, 2006) . Delap (1993) presented the quantitative results of a mark-remark experiment alongside a discussion of the shortcomings of the mark scheme as judged by the participants of the study. That study suggested a redesign to the mark scheme and a subsequent evaluation of the modifications was reported in Baird and Pinot de Moira (1997) . In the more general context of improving the reliability and validity of a test, Moskal and Leydens (2000) recognised the role of the mark scheme. They suggested that clarity might be improved by ensuring that scoring categories are well defined and that the differences between Copyright © 2014 AQA Education and its licensors. All rights reserved. AQA Education (AQA) is a registered charity (number 1073334) and a company limited by guarantee registered in England and Wales (number 3644723) . Our registered address is AQA, Devas Street, Manchester M15 6EX. score categories are clear. These ideas were extended in Pinot de Moira (2011) where the value placed on individual marks was considered. She said that: “each mark must have the same worth in order that the relative weight of items, or assessment objectives, within a paper is as intended. Equally, every mark sho

FAQs

What is the paper’s main focus regarding levels-based mark schemes?

How the rules embedded in levels-based mark schemes may influence whether measurement can be considered unbiased, from a statistical viewpoint.

How does classical test theory model observed marks?

It treats observed marks as the sum of the true mark and measurement error, assuming the error is random and uncorrelated across candidates.

What recommendation does the paper make for levels-based mark schemes?

Design levels-based mark schemes so that each band has equal width, pending further research.

Oct 20, 2026

Uploaded by Jordan Avery