Investigations of the relationship between individual differences and marker reliability - review of studies

Oct 20, 2026

Uploaded by Sarah

Review of research addressing how individual differences relate to marker reliability and how awarding bodies can predict which examiners mark most consistently versus those needing extra training or monitoring. Evidence examines links between examiner background and marking performance, including how trainee or inexperienced markers tend to be more severe and how training can reduce severity differences. Studies also consider marking inconsistency as a greater threat to reliability and report mixed findings regarding subject knowledge and teaching experience, alongside cases contrasting novice, competent, and expert markers.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Relationship between examiner background and marking performance

Examiner experience and severity

Background factors and prediction limits

Severity vs inconsistency and implications for reliability

Evidence on marking consistency and reliability

Case studies and expert vs lay markers

Centre for Education Research and Policy Investigations of the relationship between individual differences and marker reliability are crucial in determining examiner recruitment practices. A number of studies have attempted to identify factors which might allow awarding bodies to predict those examiners who are likely to mark most reliably and those who are likely to require additional training or monitoring. These studies are reviewed here. The relationship between examiner background and marking performance Research suggests that compared to experienced markers; inexperienced markers tend to mark more severely and employ different rating strategies (Ruth and Murphy, 1988; Huot, 1998; Cumming, 1990; Shohmy, Gordon and Kraemer, 1992; Weigle, 1994, 1999) . Ruth and Murphy (1988) reported a study that revealed a tendency for trainee teachers to mark essays more severely than experienced markers, though the differences were not significant. They suggested that the markersí background determined distinctly different frames of reference for judging the essays. Similarly Weigle (1999) reported that inexperienced examiners were more severe than experienced examiners. She found that prior to training, inexperienced markers could be significantly more severe than experienced markers depending on the essay title, but after training the differences in severity disappeared. She suggested that her results ìunderscore the complexity of the relationship between rater background, the scoring rubric, the prompt, and rater training in writing assessment. î (p. 171) Not all studies have replicated the relationship between inexperience and marking severity. Myford and Mislevy (1994) studied the Advanced Placement examination in Studio Art in the US. They attempted to identify background variables, including years of teaching experience, which might predict marker severity but found that the variables studied had a negligible impact on predictions of marker severity. Further, Meyer (2000a, 2000b), investigating marking in GCSE English Literature and Geography, found that length of examiner experience and a senior examinerís rating of the examinerís performance (from A - consistently excellent, to E unsatisfactory not to be re-employed) rarely proved useful as predictors of whether an examinerís marks would require adjustment to correct for severity or generosity. While there is some evidence of an association between marker experience and severity studies have failed to differentiate the effects of teaching and examining experience. Moreover, in large scale testing programmes concern is often focused on inconsistency rather than severity in marking. Variations across examiners in marking severity can be accounted for by adjusting candidatesí marks and this is common practice in UK awarding bodies (Baird and Mac, 1999) . However, mark adjustment can only be used where the examiner has been consistently severe or lenient. It is of no help when markers are inconsistent. So marking inconsistency is a much greater threat to the reliability of the marks awarded to candidates. Evidence of an association between marker background and marking consistency will now be reviewed. It is, however, ambiguous, and studies investigating this relationship have generally failed to tease out the effects of markersí subject knowledge, teaching and marking experience on marking reliability. Ecclestone (2001) carried out a case study of nine university lecturers who double-marked 45 dissertations between them over two years. Discrepancies between grades were moderated ata one-day moderation meeting, and the external examiner saw a sample of dissertations. Rough distinctions between the lecturers were made according to length of experience in assessing the programme and of other degree and Mastersí level work. The lecturers were classified as novice, competent or expert markers. Following moderation the novices had 2 fewer changes to their marks than the c

FAQs

Why are investigations into marker reliability important for examiner recruitment practices?

They determine how individual differences and rater characteristics affect reliability, helping awarding bodies decide which examiners require additional training or monitoring.

What does research suggest about inexperienced or trainee markers compared with experienced markers?

In several studies, inexperienced or trainee markers tend to mark more severely and use different rating strategies, though some differences diminish after training.

How do studies distinguish the effects of severity versus inconsistency on reliability?

Evidence indicates that mark adjustment can address consistently severe or lenient marking, but it cannot resolve inconsistencies, which pose a greater threat to reliability.

Oct 20, 2026

Uploaded by Sarah