[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-621048-105":3,"detail-sidebar-cat-0-en-105":80,"doc-detail-621048-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":73,"head_meta":75,"extra_data":77,"updated_unix":79},105,"en","do-marking-reliability-studies-have-validity-abstract","DO MARKING RELIABILITY STUDIES HAVE VALIDITY? - Abstract","","Marking reliability studies ensure examiners’ marking of high-stakes public assessments is appropriate, consistent, and fair. The paper reviews marking reliability scholarship from the early twentieth century onward, including different examination formats. It highlights a central design challenge: whether to use live or non-live examination contexts. It reports outcomes from a questionnaire completed by 89 examiners.",{"@graph":14,"@context":72},[15,34,55],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/exam/","Exam",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/do-marking-reliability-studies-have-validity-abstract/621048/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/do-marking-reliability-studies-have-validity-abstract/621048.png","ImageObject",300,407,{"name":42,"@type":43},"Angel","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-01","2026-10-20",true,{"@type":52,"interactionType":53,"userInteractionCount":26},"InteractionCounter",{"@type":54},"ViewAction",{"@type":56,"mainEntity":57},"FAQPage",[58,64,68],{"name":59,"@type":60,"acceptedAnswer":61},"Why are marking reliability studies important in high-stakes public exams?","Question",{"text":62,"@type":63},"They help ensure that examiners apply the mark scheme consistently and that marking is appropriate, consistent, and fair to candidates.","Answer",{"name":65,"@type":60,"acceptedAnswer":66},"What is the main design decision in conducting a marking reliability study?",{"text":67,"@type":63},"Choosing whether the study is conducted in a live examination setting or a non-live (controlled) context.",{"name":69,"@type":60,"acceptedAnswer":70},"How can using a non-live context affect validity?",{"text":71,"@type":63},"Non-live research enables greater control, but it raises concerns that examiners may not mark with the same diligence as they would in real marking, creating a validity threat.","https://schema.org",{"og:url":32,"og:type":74,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":76,"canonical":32},"index,follow",{"doc_id":78,"site_id":7},621048,1792464900,{"code":4,"msg":81,"data":82},"success",[83,87,91,94,99,104,109,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":84,"show_sort_weight":85,"slug":86},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":88,"show_sort_weight":89,"slug":90},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":92,"slug":93},70,"exam",{"id":95,"doc_module":4,"doc_module_name":25,"category_name":96,"show_sort_weight":97,"slug":98},5,"Comic",60,"comic",{"id":100,"doc_module":4,"doc_module_name":25,"category_name":101,"show_sort_weight":102,"slug":103},6,"Technology",50,"technology",{"id":105,"doc_module":4,"doc_module_name":25,"category_name":106,"show_sort_weight":107,"slug":108},7,"Healthcare",40,"healthcare",{"id":110,"doc_module":4,"doc_module_name":25,"category_name":111,"show_sort_weight":112,"slug":113},8,"Research & Report",30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":95,"slug":129},19,"General","general",{"code":4,"msg":81,"data":131},{"doc_id":78,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":33,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":26,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":79,"read_time":144,"preview_page_count":4},687207412472,"https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d","# DO MARKING RELIABILITY STUDIES HAVE VALIDITY?\n\nSuzanne Chamberlain  \n## ABSTRACT\n\nMarking reliability studies are essential to ensure that examiners'marking of high stakes public assessmentsis appropriate,consistent and fair to candidates.The importance of such studies is confirmed by anextensive marking reliability literature which spans the early twentieth century to the present day,and coversmany different forms of examination papers.  \nThere are many challenges in designing a marking reliability study,and perhaps the most significant of theseis deciding whether to undertake research in a live or non-live examination context.While a live context hasthe greatest external validity,it is not always feasible or fair to candidates or examiners.Alternatively,using anon-live context allows for greater control of the research environment but brings into question the validity ofthe outcomes.For example,if participants are aware that the examination scripts have no bearing oncandidates'real examination results,will they mark the research scripts with the same diligence as livescripts?  \nThis paper overviews the outcomes of a questionnaire distributed to 89 examiners who took part in acontrolled,experimental marking reliability study.The questionnaire was designed to compare the degree ofconscientiousness applied during the experimental study and under normal live marking conditions.In doingso,the questionnaire aimed to gauge the representativeness and naturalness of participants'behaviours andthus contribute to assessing the ecological validity of the primary(marking reliability)study.  \nIt is found that a small proportion of participants acknowledged that they responded differently during thestudy.It is suggested that these differences could represent a threat to he validity of the study.It isconcluded,however,that this is an inevitable drawback of controlled,non-live research,and that this iscounter-balanced by the important insights into the reliability of examiners'marking that such studies give us.  \n## INTRODUCTION\n\nThe suite of public examinations of England,Northern Ireland and Wales includes the General Certificate ofSecondary Education(GCSE),the General Certificate of Education(GCE),and a range of applied andvocational qualifications.These public examinations are high stakes and their outcomes serve severalimportant social functions.Chiefly,the examination candidates use the outcomes as ‘entry tickets'(Denscombe,2000)for other educational,training or employment opportunities.They are also used asindicators of teacher and especially school performance at national and even international levels,and tomonitor examination standards over time (Goldstein,2001).  \nGiven these important social functions,marking reliability studies have long played a crucial role ineducational research and the work of awarding bodies particularly(see for example,Meadows andBillington,2005).Such studies typically aim to explore the consistency with which examiners apply the markscheme by measuring examiners'divergence from the 'true'script scores (usually the script scores awardedby the Principal Examiner who is responsible for setting the paper and devising the accompanying markscheme).In some cases marking reliability studies aim to manipulate certain variables to explore theirimpact,while others attempt to assess the quality of operational marking(Meadows and Billington,2005).The key concern of such studies is to measure marking reliability,but in doing so some important questionsemerge relating to study design and how this impacts on the validity of the findings.  \nwww.cerp.org.uk  \n# Designing a marking reliability study\n\nOne of the key decisions to be made in designing a marking reliability study is whether the research shouldbe conducted in a live or non-live examination setting.Essentially this represents the difference between anaturalistic setting and an artificially created research environment that to a greater or lesser degreeattempts t","cbCaivBrE3uQmMRH","https://ap.wps.com/l/cbCaivBrE3uQmMRH","pdf",847536,13,"English","# ABSTRACT\n## INTRODUCTION\n# Designing a marking reliability study","[{\"question\":\"Why are marking reliability studies important in high-stakes public exams?\",\"answer\":\"They help ensure that examiners apply the mark scheme consistently and that marking is appropriate, consistent, and fair to candidates.\"},{\"question\":\"What is the main design decision in conducting a marking reliability study?\",\"answer\":\"Choosing whether the study is conducted in a live examination setting or a non-live (controlled) context.\"},{\"question\":\"How can using a non-live context affect validity?\",\"answer\":\"Non-live research enables greater control, but it raises concerns that examiners may not mark with the same diligence as they would in real marking, creating a validity threat.\"}]","DO MARKING RELIABILITY STUDIES HAVE VALIDITY? - Abstract | PDF",33]