High Stake Testing - Factors Affecting Inter-rater Reliability in Scoring of Secondary School Examination

Oct 15, 2026

The study validates factors influencing inter-rater reliability in scoring secondary school certificate (SSC) papers under high-stake testing. Urdu and English SSC papers from BISE were selected, and marking data were collected by a survey where raters scored the same papers and completed a questionnaire. A random sample of 98 raters from English and Urdu subjects was used, with analyses based on Spearman correlation to assess scoring consistency. Results show raters’ training improves inter-rater reliability, while other factors show no effect. The study recommends ensuring all raters participate in training sessions.

LanguageEnglish

CategoryExam

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Abstract

Introduction

Technical factors

Scoring criteria

Scoring scheme

Personal factors

Bulletin of Education and Research August 2020, Vol. 42, No. 2 pp. 163-179 High Stake Testing: Factors Affecting Inter-rater Reliability in Scoring of Secondary School Examination Sehar Rashid *and Nasir Mahmood** Abstract The study aimed to validate the factors that affect the inter-rater reliability of secondary school certificate (SSC) papers in high-stake testing. For this purpose, papers of Urdu and English of Board of Intermediate and Secondary Education (BISE) were selected. A survey method was used to collect marking on the same set of papers from each rater and their response to the questionnaire. The population of the study has comprised the raters for English and Urdu subjects. A sample of 98 raters was selected randomly from the list. Instruments of the study were solved papers of annual examination for both subjects and a questionnaire related to the factor that affects inter-rater reliability. According to consistency estimates of the inter-rater reliability approach, the Spearman correlation coefficient was applied to examine consistency in the scoring of raters. It was found that raters’ training effect the inter-rater reliability in their scoring. However, other factors did not affect inter-rater reliability in scoring. Based on the findings of the study, it is recommended to ensure the participation of all raters in the training session. Keywords: Inter-rater reliability, Consistency estimates of inter-rater reliability, Secondary School Examinations * Department of Education, Virtual University of Pakistan. Email: [sehar_rashid12@yahoo.com](sehar_rashid12@yahoo.com) **Professor and Dean Faculty of Education, Allama Iqbal Open University, Islamabad. [Email: mahsir1@yahoo.com](Email: mahsir1@yahoo.com) Introduction The reliability of marking is a significant aspect to ensure quality control of the assessment procedure that affects a candidate’s life chance. A major characteristic of reliability of marking is the inter-rater reliability. Inter-rater reliability is the extent to which a student obtains the same scores if different teachers scored the performance or rate the performance (Nitko, 1996). It is a quality indicator of marking reliability. It is useful to endorse the fairness of the criteria of marking and to uphold a clear understanding among raters. Consequently, it is significant to assess the inter-rater reliability in the scoring of raters to ensure marking reliability as well as the credibility of the high-stake testing. Secondary school examination is a high stake testing for every candidate as a result of these examinations provide a chance to select the career of their own choice. Results of these exams takenby BISE also provide the baseline in merit selection for high secondary institutes. In such circumstances, it is important to maintain high inter-rater reliability in the scoring of raters so that the marks of any candidate should not depend on who marked the paper. Consequently, inter-rater reliability is crucial for high stake testing. Numbers of research have been done in the nineteenth century with the concern of marking reliability which draws attention towards its importance. Research by Porterand Jelinek, (2011) found the range of inter-rater reliability from poor to moderate. It is generally considered that the subjectivity of the raters highly affects the essay type material. Therefore a measure of inter-rater reliability can be better assessed for the essay type material. Some of the researches emphasized the measure of inter-rater reliability for the essay type material of the papers. Rashid and Mahmood (2016) found moderate interrater reliability in the scoring of high-stake testing. This shows that concern of inter-rater reliability remains a subject need to be studied. All earlier studies have generated further several studies globally with the concern to assess the inter-rater reliability which also identifies different factors that influence the inte

FAQs

What was the study’s primary aim regarding high-stake testing?

To validate the factors affecting inter-rater reliability in scoring secondary school certificate (SSC) papers in high-stake testing.

How were inter-rater reliability and scoring consistency evaluated?

Raters scored the same set of Urdu and English papers, and Spearman correlation was applied to examine consistency in their scores.

Which factor was found to affect inter-rater reliability in scoring?

Raters’ training was found to affect inter-rater reliability, while other factors did not show an effect.

What practical recommendation does the study make for exam scoring teams?

Ensure the participation of all raters in the training session to improve consistency in scoring.

Oct 15, 2026

Upload by Anna Hans