Examining Rater Effects in Assessing Learners’ Speech and Writing - Insights from ICNALE

Oct 15, 2026

The study investigates rater effects in the ICNALE Global Rating Archives (GRA), using ratings from 160 raters on 140 speeches and 140 essays across 10 rating criteria. After dataset cleaning, two separate three-facet partial credit models are fitted to speech and essay data, and bias is analyzed in relation to raters’ prior rating experience. Results show six inconsistent raters in speech ratings but none in essays, and significant links between experience and specific criteria with trivial effect sizes. Implications stress rigorous rater training to improve consistency for pre-service and less-experienced in-service teachers.

LanguageEnglish

CategoryExam

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Introduction

Literature Review

ICNALE Studies Using CEFR Levels as a Proficiency Framework

English Teaching, Vol. 81, No. 1, Spring 2026, pp. 59–80 DOI: [https://doi.org/10.15858/engtea.81.1.202603.59](https://doi.org/10.15858/engtea.81.1.202603.59) [https://journal.kate.or.kr](https://journal.kate.or.kr) Examining Rater Effects in Assessing Learners’ Speech and Writing: Insights from ICNALE Hyunwoo Kim* Kim, Hyunwoo. (2026). Examining rater effects in assessing learners’ speech and writing: Insights from ICNALE. English Teaching, 81(1), 59–80. This study investigated rater effects in the ICNALE Global Rating Archives (GRA), which contain ratings from 160 raters on 140 speeches and 140 essays across 10 rating criteria. After cleaning the raw dataset, two separate three-facet partial credit models were fitted to the speech and essay datasets to examine rater consistency, and a bias analysis was conducted to investigate the relationship between raters’ prior rating experience and severity. The results indicated that six raters in the speech ratings exhibited overly inconsistent rating patterns, but no rater in the essay ratings did. It was also found that raters’ prior rating experience was significantly associated with the complexity, involvement, and accuracy criteria in the speech ratings and with the complexity criterion in the essay ratings. However, the corresponding effect sizes for these interactions were trivial, indicating limited practical impact. The educational implication is that rigorous rater training could enhance rater consistency among both pre-service and less-experienced in-service English teachers in speaking and writing assessment. Key words: ICNALE, learner corpus research, rater consistency, rating experience, Rasch measurement *Author: Hyunwoo Kim, Associate Research Fellow, Korea Institute for Curriculum and Evaluation; 8 Gyohak-ro, Deoksan-eup, Jincheon-gun, Chungcheongbuk-do 27873, Republic of Korea; [Email: hwkim@kice.re.kr](Email: hwkim@kice.re.kr) Received 31 December 2025; Reviewed 25 January 2026; Accepted 27 March 2026 © 2026 The Korea Association of Teachers of English (KATE) This is an open access article distributed under the terms of the Creative Commons Attribution License 4.0, which permits anyone to copy, redistribute, remix, transmit and adapt the work, provided the original work and source are appropriately cited. 1. INTRODUCTION Since the release of the International Corpus Network of Asian Learners of English (ICNALE), studies using this learner corpus have deepened our understanding of the developmental aspects of Asian learners of English owing to the abundance of learner data (Ishikawa, 2023b). To illustrate, the major modules ofICNALE (i.e., spoken monologues, spoken dialogues, written essays, and edited essays plus) consist of 17,458 speech and essay samples collected under controlled topics as of July 2025. Additional contextual variables such as background characteristics (e.g., gender, L1), motivation, learning history, and proficiency data have also enabled researchers to investigate the extent to which these contextual variables are associated with the quality of speeches and essays produced by Asian learners of English. In particular, among the four types of proficiency data provided in ICNALE, the Common European Framework of Reference for Languages (CEFR) levels have been primarily used as benchmarks to examine developmental aspects of Asian learners of English (Hong, 2024; Ishikawa, 2013, 2019, 2024b; Kim & Kim, 2019; Kim & Oh, 2025; Kobayashi & Abe, 2016; Strong, 2025) . However, comparatively little research has drawn on analytic rating scores from trained raters available in ICNALE even though analytic rating scores can support finer-grained analyses of developmental trajectories than CEFR levels alone (for exceptions, see Lee, 2021a, 2021b) . One plausible reason is that analytic rating scores are available for only a few subcorpora ofICNALE; another reason is that analytic ratings are susceptible to raterrelated variability. Thu

FAQs

What dataset does the study use to examine rater effects?

It uses the ICNALE Global Rating Archives (GRA), containing ratings from 160 raters on 140 speeches and 140 essays across 10 rating criteria.

How does the study model rater effects in assessing speech and writing?

It fits two separate three-facet partial credit models to the speech and essay datasets and conducts a bias analysis relating rater prior experience to rating severity.

What are the main findings about rater consistency and prior rating experience?

In speech ratings, six raters show overly inconsistent patterns, while no such rater appears in essay ratings. Prior rating experience is significantly associated with performance on certain criteria, but the corresponding effect sizes are trivial, limiting practical impact.

Oct 15, 2026

Uploaded by Emma Wilson