[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-620982-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-620982-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","testing-the-validity-of-judgements-about-geography-essays-using-the-adaptive-comparative-judgement-method-abstract","Testing the validity of judgements about geography essays using the Adaptive Comparative Judgement method - Abstract","","Adaptive comparative judgement (ACJ) is an alternative to marking in which judges compare pairs of students’ work and decide which contains more of a specified trait set. Although ACJ can yield highly reliable rank orders, prior reports do not evaluate whether judges’ decision criteria are valid. This study investigates 564 AS-level geography essays judged by 23 teachers without mark schemes, using notes after each comparison and thematic analysis to test criterion validity.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/testing-the-validity-of-judgements-about-geography-essays-using-the-adaptive-comparative-judgement-method-abstract/620982/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/testing-the-validity-of-judgements-about-geography-essays-using-the-adaptive-comparative-judgement-method-abstract/620982.png","ImageObject",300,407,{"name":42,"@type":43},"Quinn Holloway","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-03","2026-10-20",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",5,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What is adaptive comparative judgement (ACJ) and how do judges make decisions?","Question",{"text":63,"@type":64},"ACJ presents judges with pairs of students’ work and asks them to decide holistically which piece shows more of a specified trait or trait set.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How did the study test the validity of the criteria used by judges?",{"text":68,"@type":64},"Judges made notes after each decision about what was foremost in their minds. The investigation then used thematic analysis to identify and test the validity of the criteria reflected in those notes.",{"name":70,"@type":61,"acceptedAnswer":71},"What did the notes reveal about the criteria judges used?",{"text":72,"@type":64},"Overall, judges described knowledge mainly using language drawn from mark scheme wording and assessment objectives, and indirect links were found between notes and existing shared criteria, supporting criterion validity.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},620982,1792611639,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,100,105,110,114,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":97,"show_sort_weight":98,"slug":99},"Comic",60,"comic",{"id":101,"doc_module":4,"doc_module_name":25,"category_name":102,"show_sort_weight":103,"slug":104},6,"Technology",50,"technology",{"id":106,"doc_module":4,"doc_module_name":25,"category_name":107,"show_sort_weight":108,"slug":109},7,"Healthcare",40,"healthcare",{"id":111,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":112,"slug":113},8,30,"research-report",{"id":115,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":55,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":111,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":127,"language":139,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":12,"update_tm":143,"read_time":144},2336474466712,"https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd","Centre for Education Research and Policy  \nTesting the validity of judgements about geography essays using the Adaptive Comparative Judgement method  \nClaire Whitehouse  \nAbstract  \nAdaptive comparative judgement (ACJ) is an alternative to marking that presents judges with pairs of students’ work and asks them to decide, holistically, which piece of work contains more of a specified trait or set of traits. There are a number of reports on the highly reliable rank orders achieved using ACJ. However, none of these reports addresses the validity of the criteria on which judges base their decisions.  \nThe reliability of the rank ordering of 564 AS-level geography essays by 23 teachers or examiners of geography was reported previously. The judges in this study were asked to use their professional judgement when making decisions about essays; they were not provided with mark schemes or assessment objectives, but two importance statements were made available to them. After each judgement (92 .4% of the total), the judges in this empirical study made notes about what was to the forefront of their minds when they made a decision between two essays. The investigation reported here uses thematic analysis of these notes to identify and test the validity of the criteria the judges used to make their decisions.  \nOn the whole the judges used the language of the mark scheme and the assessment objectives when describing the knowledge demonstrated in the essays. They used language from these two sources to a lesser extent when describing skills, but nonetheless indirect links could be established between the content of the notes and existing documents. These links demonstrate the use of existing shared criteria by the judges, thus the validity of the criteria used by judges in their decision-making was confirmed. However, these criteria are already established as part of examiner training, marking and teacher support. This has implications for the introduction of ACJ as a replacement for marking, which are discussed.  \nKey words: adaptive comparative judgement, psychological validity, paired comparisons, shared criteria  \nIntroduction  \nPurpose  \nA recent empirical study investigated the feasibility of using adaptive comparative judgement (ACJ) as an alternative to marking in national high stakes assessments (Whitehouse & Pollitt, 2012) . One of the outcomes of the study was to place 564 essay responses to an AS-level geography paper in a rank order with a reliability of 0.97, which is high when compared with thereliabilities achieved in traditional marking. The rank order was based on quality parameters that were estimated using the Rasch model to fit the decisions from a number of paired comparisons of the essays. The paired comparisons were made by 23 judges who were teachers or examiners of geography.  \n[www.cerp.org.uk](www.cerp.org.uk)  \nCopyright © 2013 AQA Education and its licensors. All rights reserved.  \nAQA Education (AQA) is a registered charity (number 1073334) and a company limited by guarantee registered in England and Wales (number 3644723) . Our registered address is AQA, Devas Street, Manchester M15 6EX.  \nOne of the judges’ tasks in the study was to make a note after each decision as to what they were thinking about when they made their decision. It is the analysis of the content of these notes that is reported here as a way of testing the validity of the judgements. The validity under test is that referred to by Bramley (2007) as the “psychological validity” or whether it is possible for judges making paired comparisons of essays to observe sufficient and appropriate construct-relevant features on which to base their judgements.  \nThe aim of the current investigation was to explore the following research questions.  \n􀁸 What features of the essays are influencing the judges’ decisions in adaptive comparative judgement?  \n􀁸 Is it possible to relate the source or sources of these influences to existing criteria?  \nPaired comp","cbCaihPsJs0OkY58","https://ap.wps.com/l/cbCaihPsJs0OkY58","pdf",179187,"English","# Introduction\n## Purpose\n## Paired comparisons and adaptive comparative judgement (ACJ)","[{\"question\":\"What is adaptive comparative judgement (ACJ) and how do judges make decisions?\",\"answer\":\"ACJ presents judges with pairs of students’ work and asks them to decide holistically which piece shows more of a specified trait or trait set.\"},{\"question\":\"How did the study test the validity of the criteria used by judges?\",\"answer\":\"Judges made notes after each decision about what was foremost in their minds. The investigation then used thematic analysis to identify and test the validity of the criteria reflected in those notes.\"},{\"question\":\"What did the notes reveal about the criteria judges used?\",\"answer\":\"Overall, judges described knowledge mainly using language drawn from mark scheme wording and assessment objectives, and indirect links were found between notes and existing shared criteria, supporting criterion validity.\"}]","Testing the validity of judgements about geography essays using the Adaptive Comparative Judgement method - Abstract | PDF",1792464822,48]