The effect of specialisms on the marking reliability of an English literature examination

Oct 20, 2026

The report examines whether examiner specialisms improve marking reliability in an English literature high-stakes examination. Using multilevel modelling of mark and remark data from quality monitoring samples, it analyzes 53 examiners’ performance on more than 3,000 responses to optional questions tied to prescribed texts. Examiners’ specialisms are defined by texts associated with their home centre. The study concludes that examiners with experience across a broader range of specialisms mark more reliably, with statistical significance, and outlines directions for further research.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Summary

Background

Centre for Education Research and Practice Who is the specialist? The effect of specialisms on the marking reliability of an English literature examination Claire Whitehouse Summary The effect of examiner specialisms on marking reliability is an under-researched topic. This report explores the topic using multilevel modelling of mark remark data from quality monitoring samples. It aims to provide an answer to the question: do examiners mark more accurately when they mark responses from their own specialisms? Data from 53 examiners across more than 3,000 responses to optional questions from an Alevel English literature examination were modelled. The optionality in the question paper was based on prescribed texts. Each examiner was matched to a home centre. The texts offered by a home centre (based on candidates’ choices in the examination) became the examiner’s specialisms. The report draws one main conclusion. Examiners who have experience within their working environment of a range of specialisms, mark more reliably than those examiners who work with a limited number of specialisms. This finding was statistically significant. Suggestions for future work in this area are provided at the end of the report. Keywords: marking reliability, examiner specialisms, optional essays, English literature Background Recently, the accuracy of the marking of high stakes examinations, such as GCSEs and Alevels, has been the focus of attention for a number of stakeholders including government (na, 2012a), head teachers (na, 2012b), and the regulator (Opposs & He, 2011), as well as awarding organisations. Inaccurate marking compromises the validity of qualifications. Awarding organisations, operating in an environment of continuous improvement and transparency, must secure validity. It is, therefore, incumbent on them to consider all means available to increase the reliability of marking. To do this, it is necessary to identify and understand the variables that affect marking reliability. To this end Suto and Nádas (2008) proposed a qualitative model with two main influences on marking reliability: examiners’ expertise and the demands of the marking task. They identified a number of aspects to examiners’ expertise including: marking experience, teaching experience, level of general education, subject knowledge, personality traits and training. Suto and coworkers (2008 & 2011) used multiple mark data for selected questions from GCSEs in mathsand science to investigate selection criteria. The most pertinent finding from their studies was that highest level of education, regardless of the subject area, was a better predictor of marking accuracy than highest level of education in a relevant subject, previous marking experience or teaching experience. Royal-Dawson and Baird (2009) arrived at a similar conclusion in the context of the marking of Key Stage 3 English. They recommended that, except for a few highly subject specific items, the requirement of teaching experience be loosened to allow nonteaching graduates of any subject to mark responses as their marking reliability was at least as [www.cerp.org.uk](www.cerp.org.uk) Copyright © 2014 AQA Education and its licensors. All rights reserved. AQA Education (AQA) is a registered charity (number 1073334) and a company limited by guarantee registered in England and Wales (number 3644723) . Our registered address is AQA, Devas Street, Manchester M15 6EX. good as that of those examiners with teaching experience. In a designed study using selected questions from a GCSE English examination, Meadows and Billington (2010) attempted to control for marking experience, teaching experience and subject knowledge. They concluded that trainee English teachers, undergraduates of English and undergraduates of other subjects could mark as accurately as GCSE English examiners, provided they were offered the same level of training. Some questions requiring more complex marking strategies w

FAQs

What question does the report aim to answer about marking reliability?

It asks whether examiners mark more accurately when marking responses from their own specialisms.

How is “examiner specialism” defined in the study?

Specialisms are identified from the texts offered by each examiner’s home centre, based on candidates’ choices in the examination.

What main conclusion does the report reach?

Examiners with experience across a wider range of specialisms in their working environment mark more reliably than those with limited specialisms, and this result is statistically significant.

Oct 20, 2026

Uploaded by Riley West