[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-582847-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-582847-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","ceme-technical-report-examining-inter-rater-reliability-of-evaluators-judging-teacher-performance-proposing-an-alternative-to-cohens-kappa","CEME Technical Report - Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance - Proposing an Alternative to Cohen’s Kappa","","Examining inter-rater reliability for teacher performance ratings raises validity concerns for Kappa when category prevalence is low and agreement between raters is high. The study proposes the Lambda Coefficient of Rater-Mediated Agreement to address chance agreement under assumptions about rater tendencies and the rater-mediated assessment process. Empirical results from teacher profile ratings confirm Kappa’s shortcomings and show improved robustness for Lambda-1, Lambda-2, Gwet’s AC1, and Gwet’s AC2 across 65 raters. Simulations support Lambda’s robustness under Kappa-problematic conditions.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/ceme-technical-report-examining-inter-rater-reliability-of-evaluators-judging-teacher-performance-proposing-an-alternative-to-cohens-kappa/582847/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/ceme-technical-report-examining-inter-rater-reliability-of-evaluators-judging-teacher-performance-proposing-an-alternative-to-cohens-kappa/582847.png","ImageObject",300,407,{"name":42,"@type":43},"Rhys","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-31","2026-10-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",7,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why is Cohen’s Kappa questioned for teacher performance ratings?","Question",{"text":63,"@type":64},"Kappa validity is questioned when rating-scale category prevalence is low and raters show high agreement, which can limit how well Kappa corrects for chance agreement.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"What alternative does the paper propose to address Kappa’s concerns?",{"text":68,"@type":64},"The paper proposes the Lambda Coefficient of Rater-Mediated Agreement, designed to correct for chance agreement based on assumptions about raters and the rater-mediated assessment process.",{"name":70,"@type":61,"acceptedAnswer":71},"How did the study evaluate the performance of the alternative coefficients?",{"text":72,"@type":64},"It used actual ratings from an inter-rater reliability exercise and additional simulation data, comparing robustness across problematic conditions for Kappa and assessing variability across the 65 raters.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},582847,1792167351,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,110,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":107,"show_sort_weight":108,"slug":109},"Healthcare",40,"healthcare",{"id":111,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":112,"slug":113},8,30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":111,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":108,"language":139,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":12,"update_tm":143,"read_time":144,"preview_page_count":4},687207024643,"https://ap-avatar.wpscdn.com/davatar_3d24733baf745e90a7e4bdd5f77d97b2","| CEMETR-2022-02 February 2022 | CEME\u003Cbr>Technical Report |\n| --- | --- |\n|  |  |\n| The Center for | Educational Measurement and Evaluation |\n| A PUBLICATION OF THE CENTER FOR EDUCATIONAL MEASUREMENT AND EVALUATION | Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance: Proposing an Alternative to Cohen’s Kappa\u003Cbr>Richard Lambert\u003Cbr>T. Scott Holcomb Bryndle Bottoms\u003Cbr>|\n\nExamining Inter-Rater Reliability of Evaluators Judging Teacher Performance: Proposing an Alternative to Cohen’s Kappa  \nRichard G. Lambert  \nCenter for Educational Measurement and Evaluation  \nUNC Charlotte  \nT. Scott Holcomb  \nDuke University  \nBryndle Bottoms  \nUniversity of South Carolina  \nAn earlier version of this paper was presented to the virtual Annual Meeting of the National Council on Measurement in Education, June, 2021.  \nAbstract  \nThe validity of the Kappa coefficient of chance-corrected agreement has been questioned when the prevalence of specific rating scale categories is low and agreement between raters is high. The researchers proposed the Lambda Coefficient ofRater-Mediated Agreement as an alternative to Kappa to address these concerns. Lambda corrects for chance agreement based on specific assumptions about raters and the rater-mediated assessment process including raterspecific tendencies for strict or lenient ratings. Actual ratings of teacher profiles from an interrater reliability exercise confirmed the shortcomings of Kappa. The rater data also demonstrated the robustness of Lambda-1, Lambda-2, Gwet’s AC1, and Gwet’s AC2 to the data conditions that are problematic for Kappa. All four alternative chance-corrected agreement coefficients showed less variability across the 65 raters than Kappa. However, AC-2 was undetermined for 57 of the 65 raters. Simulation data demonstrated the robustness of the Lambda Coefficient of  \nRater-Mediated Agreement to the data conditions that are problematic for Kappa.  \nExamining Inter-Rater Reliability of Evaluators Judging Teacher Performance:  \nAn Alternative to Cohen’s Kappa  \nCohen’s Kappa (Cohen, 1960) and Weighted Kappa (Cohen, 1968) are widely used measures of chance-corrected agreement between raters. Researchers have raised questions about whether Kappa is actually correcting for chance agreement and whether it is useful for identifying and separating various sources of disagreement. The validity of Kappa coefficients has been questioned when prevalence of specific categories on a rating scale is low and agreement is high. In addition, researchers have raised questions about the generalizability of Kappa coefficients across populations and study conditions (Cicchetti & Feinstein, 1990; Feinstein & Cicchetti; 1990; Thompson & Walter, 1988) . Gwet (2008) introduced AC1 and AC2 as alternatives to Kappa and demonstrated they are robust indexes not susceptible to the identified shortcomings of Kappa.  \nThis study proposed an alternative to Kappa, Weighted Kappa, AC1, and AC2 that is rooted in theory regarding rated-mediated assessment (Engelhard & Wind, 2018) . We designed the Lambda Coefficient ofRater-Mediated Agreement for use with the ordinal scales often used to evaluate teacher performance. It examines inter-rater agreement corrected for the probability that raters may agree with expert raters by chance due to the response process they employ when they are uncertain about how to place a teacher on a rubric.  \nRater-Mediated Assessment Theory  \nMost research on rater cognition focuses on the mental processes used by raters of student or examinee performance. The rating process followed by raters judging constructed responses is intricate; however, the judgment of teacher performance can pose even greater complexities. Similar to other examples ofrater-mediated assessments, an observers’ level of  \nexpertise, and the overall scoring task demands, can influence ratings of teacher performance (Bell et al., 2018; Suto, 2012). Understanding rater cognition is crucial to makin","cbCaiqiLaf1uP24D","https://ap.wps.com/l/cbCaiqiLaf1uP24D","pdf",939969,"English","# Abstract\n# Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance\n## Cohen’s Kappa and alternative chance-corrected agreement coefficients\n# Rater-Mediated Assessment Theory\n## Standards and evidence for cognitive processes\n## Lens Model and rater judgment decision-making","[{\"question\":\"Why is Cohen’s Kappa questioned for teacher performance ratings?\",\"answer\":\"Kappa validity is questioned when rating-scale category prevalence is low and raters show high agreement, which can limit how well Kappa corrects for chance agreement.\"},{\"question\":\"What alternative does the paper propose to address Kappa’s concerns?\",\"answer\":\"The paper proposes the Lambda Coefficient of Rater-Mediated Agreement, designed to correct for chance agreement based on assumptions about raters and the rater-mediated assessment process.\"},{\"question\":\"How did the study evaluate the performance of the alternative coefficients?\",\"answer\":\"It used actual ratings from an inter-rater reliability exercise and additional simulation data, comparing robustness across problematic conditions for Kappa and assessing variability across the 65 raters.\"}]","CEME Technical Report - Examining Inter-Rater Reliability of Evaluators Judging Teacher Performance - Proposing an Alternative to Cohen’s Kappa | PDF",1792044664,101]