CEME Technical Report - Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance - Proposing an Alternative to Cohen’s Kappa

Oct 15, 2026

Uploaded by Rhys

Examining inter-rater reliability for teacher performance ratings raises validity concerns for Kappa when category prevalence is low and agreement between raters is high. The study proposes the Lambda Coefficient of Rater-Mediated Agreement to address chance agreement under assumptions about rater tendencies and the rater-mediated assessment process. Empirical results from teacher profile ratings confirm Kappa’s shortcomings and show improved robustness for Lambda-1, Lambda-2, Gwet’s AC1, and Gwet’s AC2 across 65 raters. Simulations support Lambda’s robustness under Kappa-problematic conditions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages40

Reading Time1 h 41 min

AccessFree to download

Abstract

Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance

Cohen’s Kappa and alternative chance-corrected agreement coefficients

Rater-Mediated Assessment Theory

Standards and evidence for cognitive processes

Lens Model and rater judgment decision-making

| CEMETR-2022-02 February 2022 | CEME
Technical Report | | --- | --- | | | | | The Center for | Educational Measurement and Evaluation | | A PUBLICATION OF THE CENTER FOR EDUCATIONAL MEASUREMENT AND EVALUATION | Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance: Proposing an Alternative to Cohen’s Kappa
Richard Lambert
T. Scott Holcomb Bryndle Bottoms
| Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance: Proposing an Alternative to Cohen’s Kappa Richard G. Lambert Center for Educational Measurement and Evaluation UNC Charlotte T. Scott Holcomb Duke University Bryndle Bottoms University of South Carolina An earlier version of this paper was presented to the virtual Annual Meeting of the National Council on Measurement in Education, June, 2021. Abstract The validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient ofRater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance agreement based on specific assumptions about raters and the rater-mediated assessment process including raterspecific tendencies for strict or lenient ratings. Actual ratings of teacher profiles from an interrater reliability exercise confirmed the shortcomings of Kappa. The rater data also demonstrated the robustness of Lambda-1, Lambda-2, Gwet’s AC1, and Gwet’s AC2 to the data conditions that are problematic for Kappa. All four alternative chance-corrected agreement coefficients showed less variability across the 65 raters than Kappa. However, AC-2 was undetermined for 57 of the 65 raters. Simulation data demonstrated the robustness of the Lambda Coefficient of Rater-Mediated Agreement to the data conditions that are problematic for Kappa. Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance: An Alternative to Cohen’s Kappa Cohen’s Kappa (Cohen, 1960) and Weighted Kappa (Cohen, 1968) are widely used measures of chance-corrected agreement between raters. Researchers have raised questions about whether Kappa is actually correcting for chance agreement and whether it is useful for identifying and separating various sources of disagreement. The validity of Kappa coefficients has been questioned when prevalence of specific categories on a rating scale is low and agreement is high. In addition, researchers have raised questions about the generalizability of Kappa coefficients across populations and study conditions (Cicchetti & Feinstein, 1990; Feinstein & Cicchetti; 1990; Thompson & Walter, 1988) . Gwet (2008) introduced AC1 and AC2 as alternatives to Kappa and demonstrated they are robust indexes not susceptible to the identified shortcomings of Kappa. This study proposed an alternative to Kappa, Weighted Kappa, AC1, and AC2 that is rooted in theory regarding rated-mediated assessment (Engelhard & Wind, 2018) . We designed the Lambda Coefficient ofRater-Mediated Agreement for use with the ordinal scales often used to evaluate teacher performance. It examines inter-rater agreement corrected for the probability that raters may agree with expert raters by chance due to the response process they employ when they are uncertain about how to place a teacher on a rubric. Rater-Mediated Assessment Theory Most research on rater cognition focuses on the mental processes used by raters of student or examinee performance. The rating process followed by raters judging constructed responses is intricate; however, the judgment of teacher performance can pose even greater complexities. Similar to other examples ofrater-mediated assessments, an observers’ level of expertise, and the overall scoring task demands, can influence ratings of teacher performance (Bell et al., 2018; Suto, 2012). Understanding rater cognition is crucial to makin

FAQs

Why is Cohen’s Kappa questioned for teacher performance ratings?

Kappa validity is questioned when rating-scale category prevalence is low and raters show high agreement, which can limit how well Kappa corrects for chance agreement.

What alternative does the paper propose to address Kappa’s concerns?

The paper proposes the Lambda Coefficient of Rater-Mediated Agreement, designed to correct for chance agreement based on assumptions about raters and the rater-mediated assessment process.

How did the study evaluate the performance of the alternative coefficients?

It used actual ratings from an inter-rater reliability exercise and additional simulation data, comparing robustness across problematic conditions for Kappa and assessing variability across the 65 raters.

Oct 15, 2026

Uploaded by Rhys