[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-581104-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-581104-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","optimizing-inter-rater-reliability-in-foreign-language-constructed-response-assessments","OPTIMIZING INTER-RATER RELIABILITY IN FOREIGN LANGUAGE CONSTRUCTED-RESPONSE ASSESSMENTS","","This study examines inter-rater reliability in a constructed-response English proficiency test developed by the Foreign Language Assessment Program (PELEx) in Costa Rica. Thirty university instructors completed three writing tasks aligned with A2, B2, and C1 CEFR bands, each scored by two trained raters. Percent agreement, Cohen’s weighted kappa, intraclass correlation coefficients (ICCs), and Generalizability Theory (G-Theory) are used to estimate reliability. While traditional estimators indicate good reliability, G-Theory shows additional error sources, especially prompt-related variability, which impacts score comparability and fairness. Results suggest that prompt calibration and structured scoring protocols across proficiency levels can strengthen reliability, particularly in high-stakes contexts.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/optimizing-inter-rater-reliability-in-foreign-language-constructed-response-assessments/581104/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/optimizing-inter-rater-reliability-in-foreign-language-constructed-response-assessments/581104.png","ImageObject",300,407,{"name":42,"@type":43},"Rowan","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-15","2026-10-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",10,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What is the main focus of the study?","Question",{"text":63,"@type":64},"The study focuses on optimizing inter-rater reliability in foreign-language constructed-response assessments used in an English proficiency test in Costa Rica.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How is inter-rater reliability estimated?",{"text":68,"@type":64},"Inter-rater reliability is estimated using percent agreement, Cohen’s weighted kappa, intraclass correlation coefficients (ICCs), and Generalizability Theory (G-Theory).",{"name":70,"@type":61,"acceptedAnswer":71},"What does G-Theory add beyond traditional reliability estimators?",{"text":72,"@type":64},"G-Theory reveals additional sources of error not captured well by kappa or ICC, particularly prompt-related variability and person-by-prompt interaction effects that can exceed 20% of total variance in the C1 task.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},581104,1792189344,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":55,"slug":125},"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":113,"preview_page_count":4},1099514067415,"https://ap-avatar.wpscdn.com/avatar/100002539d78ffe74a7?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779092875211072502","Research in Pedagogy, Vol. 15, No. 2, Year 2025, pp. 471-482  \nJose Fabian Elizondo-Gonzalez1 Original scientific paper  \nUniversidad de Costa Rica UDC: 37.022  \n[http://doi.org/10.5937/IstrPed2502471F](http://doi.org/10.5937/IstrPed2502471F)  \n==================================================================================  \nOPTIMIZING INTER-RATER RELIABILITY IN FOREIGN LANGUAGE CONSTRUCTED-RESPONSE ASSESSMENTS  \nAbstract: This study examines inter-rater reliability in a constructed-response English proficiency test developed by the Foreign Language Assessment Program (PELEx) in Costa Rica. Thirty university instructors completed three writing tasks aligned with A2, B2, and C1 CEFR bands, each scored by two trained raters. Inter-rater reliability was estimated using percent agreement, Cohen’s weighted kappa, intraclass correlation coefficients (ICCs), and Generalizability Theory (G-Theory). While traditional estimators suggested good to excellent reliability, G-Theory revealed additional sources of error not accounted for by kappa or ICC, particularly promptrelated variability. For example, in the C1 task, person-by-prompt interaction accounted for over 20% of total variance. These findings suggest that while rater training remains important, prompt-related variability must also be addressed to ensure fairness and score comparability.  \nIncorporating calibrated prompts and structured scoring protocols across proficiency levels may strengthen the reliability of constructed-response tasks, especially in high-stakes settings.  \nKeywords: constructed-response assessment, Costa Rica, EFL, language assessment, reliability  \nIntroduction  \nConstructed-response2 assessment deals with values assigned to observable outcomes using measurement tools, such as rubrics or checklists (Scott, 2003) . Unlike traditional norm-referenced assessments, constructed-response assessments are criterion-referenced, comparing test-takers’performance against predetermined standards rather than to other test-takers. However, for such assessments to be meaningful and equitable, they must not only align with construct-relevant criteria but also provide empirical evidence of their reliability and validity (Burger & Burger, 2005; Wagner, 2020) .  \nReliability is crucial in constructed-response assessment, as test-taker behaviors are rated by judges, something that can introduce multiple sources of error in the scoring process. Reliability, or the consistency/precision in test scores (Revelle & Condon, 2019), can be compromised when judges are not properly trained or calibrated for specific scoring tasks, not adhering to scoring protocols or being too lenient or severe (Hallgren, 2012; Lyness et al., 2021). Also, raters can suffer from a ‘halo’effect, where test-taker performance is rated based on impressions and not established protocols, introducing judgement bias (Landy & Sigall, 1974; Wetzel et al., 1981) . What is more, rater fatigue can affect the reliability of scoring procedures, in that it has been shown that test-taker performance is more severely rated for the first tasks in a testing batch, and last tasks are more leniently rated (Mahshanian & Shahnazari, 2020; Wagner, 2020) . These sources of error variance pose a serious threat to the construct validity of constructed-response assessments, as they may distort score interpretations and ultimately undermine test fairness.  \n[1](1 josefabian.elizondo@ucr.ac.cr)[ ](1 josefabian.elizondo@ucr.ac.cr)[josefabian.elizondo@ucr.ac.cr](1 josefabian.elizondo@ucr.ac.cr); [https://orcid.org/0000-0003-4819-0213](https://orcid.org/0000-0003-4819-0213)  \n2 In this study, the terms “constructed-response” and “performance-based” will be considered interchangeable.  \nResearch in Pedagogy, Vol. 15, No. 2, Year 2025, pp. 471-482  \nMeasuring Reliability in Constructed-response Assessment  \nThough there are multiple methods to account for test reliability, the most-commonly used techniques have been designed to gather ","cbCaiiJoPOCzaquz","https://ap.wps.com/l/cbCaiiJoPOCzaquz","pdf",210743,12,"English","# Introduction\n## Measuring Reliability in Constructed-response Assessment","[{\"question\":\"What is the main focus of the study?\",\"answer\":\"The study focuses on optimizing inter-rater reliability in foreign-language constructed-response assessments used in an English proficiency test in Costa Rica.\"},{\"question\":\"How is inter-rater reliability estimated?\",\"answer\":\"Inter-rater reliability is estimated using percent agreement, Cohen’s weighted kappa, intraclass correlation coefficients (ICCs), and Generalizability Theory (G-Theory).\"},{\"question\":\"What does G-Theory add beyond traditional reliability estimators?\",\"answer\":\"G-Theory reveals additional sources of error not captured well by kappa or ICC, particularly prompt-related variability and person-by-prompt interaction effects that can exceed 20% of total variance in the C1 task.\"}]","OPTIMIZING INTER-RATER RELIABILITY IN FOREIGN LANGUAGE CONSTRUCTED-RESPONSE ASSESSMENTS | PDF",1792038856]