[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-620863-105":3,"detail-sidebar-cat-0-en-105":80,"doc-detail-620863-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":73,"head_meta":75,"extra_data":77,"updated_unix":79},105,"en","identifying-errant-markers-quality-assurance-systems-in-an-e-marking-environment","Identifying Errant Markers: Quality Assurance Systems in an E-Marking Environment","","The paper investigates how innovative technology can support robust quality assurance systems for marking reliability in high-stakes national qualifications. Using Monte Carlo simulation, it compares a hierarchical seeded quality assurance model with a peer-pair double marked model to assess how item sharing affects systematic marker differences and paper-level reliability. It concludes that sampling-based elements mainly function to identify errant markers for retraining, and that peer-pair double marking is more effective at detecting errant markers under the same parameter settings. It also reviews the practical rationale and evidence for e-marking reliability and validity.",{"@graph":14,"@context":72},[15,34,55],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/identifying-errant-markers-quality-assurance-systems-in-an-e-marking-environment/620863/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/identifying-errant-markers-quality-assurance-systems-in-an-e-marking-environment/620863.png","ImageObject",300,407,{"name":42,"@type":43},"Riley West","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-03","2026-10-20",true,{"@type":52,"interactionType":53,"userInteractionCount":30},"InteractionCounter",{"@type":54},"ViewAction",{"@type":56,"mainEntity":57},"FAQPage",[58,64,68],{"name":59,"@type":60,"acceptedAnswer":61},"What simulation method is used to study the quality assurance systems for e-marking?","Question",{"text":62,"@type":63},"The paper uses Monte Carlo simulation to explore and compare two quality assurance models: a hierarchical seeded system and a peer-pair double marked system.","Answer",{"name":65,"@type":60,"acceptedAnswer":66},"How does the peer-pair double marked system compare with the hierarchical seeded system for detecting errant markers?",{"text":67,"@type":63},"For a given set of quality assurance parameters, the peer-pair double marking system is more likely to identify an errant marker than the hierarchical seeded system.",{"name":69,"@type":60,"acceptedAnswer":70},"What is the practical role of a quality assurance system that includes sampling?",{"text":71,"@type":63},"The study concludes that sampling elements have little potential to influence marking reliability directly; instead, they mainly serve to identify errant markers for retraining.","https://schema.org",{"og:url":32,"og:type":74,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":76,"canonical":32},"index,follow",{"doc_id":78,"site_id":7},620863,1792590102,{"code":4,"msg":81,"data":82},"success",[83,87,91,95,100,105,110,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":84,"show_sort_weight":85,"slug":86},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":88,"show_sort_weight":89,"slug":90},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":92,"show_sort_weight":93,"slug":94},"Exam",70,"exam",{"id":96,"doc_module":4,"doc_module_name":25,"category_name":97,"show_sort_weight":98,"slug":99},5,"Comic",60,"comic",{"id":101,"doc_module":4,"doc_module_name":25,"category_name":102,"show_sort_weight":103,"slug":104},6,"Technology",50,"technology",{"id":106,"doc_module":4,"doc_module_name":25,"category_name":107,"show_sort_weight":108,"slug":109},7,"Healthcare",40,"healthcare",{"id":111,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":112,"slug":113},8,30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":96,"slug":129},19,"General","general",{"code":4,"msg":81,"data":131},{"doc_id":78,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":111,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":30,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":145},1099523885074,"https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc","Centre for Education Research and Policy  \nIdentifying Errant Markers: Quality Assurance Systems in an E  \nMarking Environment  \nAnne Pinot de Moira  \nThe exploitation of innovative technology in the administration of high stakes national qualifications, has provided the opportunity to develop robust quality assurance systems to monitor and improve marking reliability. This paper uses Monte Carlo simulation to explore two simple models of quality assurance: a hierarchical seeded system and a peerpair double marked system. The models suggest that item marking minimises the effect of systematic differences between markers but the extent to which this improves reliability at a paper level is dependent upon the number of items on a paper. Beyond the positive effect of dividing a single paper among different markers, it is concluded that a quality assurance system which includes any element of sampling has little potential to influence marking reliability directly. It becomes a system for identifying errant markers for retraining. It is shown that, for a given set of quality assurance parameters, the peerpair double marking system is more likely to identify an errant marker than the  \nhierarchical seeded system.  \nKeywords: hierarchical seeded quality assurance system; peer-pair double marked quality assurance system; marking tolerance; e-marking.  \nThe Practicalities of E-Marking  \nThe potential to use innovative technology in education and assessment has been recognised for many years. Initially the focus was on teaching and learning in the classroom (see, for example, Sheingold, Kane, Endreweit, & Billings, 1981; Sheingold & Tucker, 1990) but the value for assessment provision and administration was quickly embraced (Means et al., 1993) . Since its infancy at the turn of the century, the use of technology in the provision of high stakes national examinations in the United Kingdom has proliferated. This has been particularly apparent in the area of on-screen marking, or e-marking, where candidate responses are presented to a marker electronically and that marker never has sight of the original paper version.  \nWith the rapid advances has come a need to ensure the maintenance of assessment reliability and validity. Studies have seemed to confirm that e-marking provides a satisfactory substitute for paper-based marking, although findings have been limited by the experimental design. The validation of e-marking systems has been necessarily coincident with the development of new technology. Therefore the  \nextent to which these studies have successfully replicated a live marking environment is restricted and, moreover, the published evaluation of operational quality assurance systems is largely absent. Nevertheless, Fowles (2002) showed that there was a close relationship between marks awarded by chemistry markers using conventional paperbased marking and those awarded using e-marking. Further evidence was presented in Powers, Farnum, Grant, & Kubota (1997), Whetton & Newton (2002) and Sturman & Kispal (2003), all of whom reported a high correlation between the outcomes of conventional and on-screen marking.  \nOnce assured that marking is unimpaired by the electronic environment, the advantages of e-marking are understood to be numerous. Whetton & Newton (2002), Hudson (2009) and Williams & van Lent (2002) all highlighted potential benefits. They can be paraphrased thus:  \n1) marking would be speeded up, as papers would spend less time in transit;  \n2) data collection would become entirely automated, eliminating the inevitable errors that creep into manual processes;  \n3) marking might become more accurate, as markers could focus upon specific questions, termed items, rather than having to mark all items from an examination;  \n4) items that do not require subject matter expertise to mark could be marked by non-experts, thereby using human resources far more effectively;  \n5) marking might become less biased, as the introduction of ra","cbCaifD8VEAPhW2X","https://ap.wps.com/l/cbCaifD8VEAPhW2X","pdf",826572,27,"English","# Identifying Errant Markers: Quality Assurance Systems in an E-Marking Environment\n## Monte Carlo simulation and comparison of QA models\n## Implications for identifying errant markers and retraining\n# The Practicalities of E-Marking\n## Reasons for adopting e-marking\n## Reliability, validity, and evidence from prior studies","[{\"question\":\"What simulation method is used to study the quality assurance systems for e-marking?\",\"answer\":\"The paper uses Monte Carlo simulation to explore and compare two quality assurance models: a hierarchical seeded system and a peer-pair double marked system.\"},{\"question\":\"How does the peer-pair double marked system compare with the hierarchical seeded system for detecting errant markers?\",\"answer\":\"For a given set of quality assurance parameters, the peer-pair double marking system is more likely to identify an errant marker than the hierarchical seeded system.\"},{\"question\":\"What is the practical role of a quality assurance system that includes sampling?\",\"answer\":\"The study concludes that sampling elements have little potential to influence marking reliability directly; instead, they mainly serve to identify errant markers for retraining.\"}]","Identifying Errant Markers: Quality Assurance Systems in an E-Marking Environment | PDF",1792464695,68]