[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-594675-105":59,"doc-detail-594675-en":134},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":127,"head_meta":129,"extra_data":131,"updated_unix":133},105,"en","a-ustralian-universities-review-when-rating-systems-do-not-rate-evaluating-eras-performance","A USTRALIAN UNIVERSITIES’ REVIEW - When rating systems do not rate - Evaluating ERA’s performance","","This paper analyzes the Australian Government’s Excellence in Research for Australia (ERA) 2015 outcomes in which rating systems did not rate 1.5% of university research submissions. It examines public claims about gaming or “coding errors” and argues unrated results also stem from evaluation practices within Psychology and the Medical and Health Sciences Research Evaluation Committee. It highlights governance failures—non-appealability and limited transparency—and closes with recommendations to strengthen ERA policy and procedures to build future trust.",{"@graph":69,"@context":126},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/a-ustralian-universities-review-when-rating-systems-do-not-rate-evaluating-eras-performance/594675/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/a-ustralian-universities-review-when-rating-systems-do-not-rate-evaluating-eras-performance/594675.png","ImageObject",300,407,{"name":92,"@type":93},"Ben ","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-25","2026-10-15",true,{"@type":102,"interactionType":103,"userInteractionCount":19},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118,122],{"name":109,"@type":110,"acceptedAnswer":111},"What does the paper investigate about ERA 2015?","Question",{"text":112,"@type":113},"The paper investigates why rating systems failed to rate a portion of university submissions in ERA 2015, focusing on both reported coding-error explanations and other evaluation practices.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How does ERA rate research indicators and what new outcome appeared in 2015?",{"text":117,"@type":113},"ERA rates disciplines from 1 to 5 and uses “not assessed” when no submission is made; in 2015 it also introduced “not rated” (n/r) for disciplinary areas where committees could not assign a numerical rating due to coding errors.",{"name":119,"@type":110,"acceptedAnswer":120},"Why does the paper argue that unrated outcomes raise governance concerns?",{"text":121,"@type":113},"It notes that unrated outcomes are high-stakes and the evaluation process breaches public administration principles by being not appealable and lacking appropriate transparency.",{"name":123,"@type":110,"acceptedAnswer":124},"What are the paper’s main recommendations?",{"text":125,"@type":113},"The paper concludes with recommendations aimed at strengthening ERA policy and procedures to enhance confidence and credibility in future ERA rounds.","https://schema.org",{"og:url":83,"og:type":128,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":130,"canonical":83},"index,follow",{"doc_id":132,"site_id":62},594675,1792170839,{"code":4,"msg":5,"data":135},{"doc_id":132,"user_id":136,"nickname":92,"user_avatar":137,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":138,"file_id":139,"file_url":140,"file_type":141,"file_size":142,"view_count":19,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":143,"language":144,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":145,"faqs":146,"seo_title":147,"seo_description":67,"update_tm":148,"read_time":149},2336478951081,"https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c","A U S T R A L I A N U N I V E R S I T I E S ’ R E V I E W  \nWhen rating systems do not rate  \nEvaluating ERA’s performance  \nPaul Henman  \nUniversity of Queensland  \nScott D Brown & Simon Dennis  \nUniversity of Newcastle  \nIn 2015, the Australian Government’s Excellence in Research for Australia (ERA) assessment of research quality declined to rate 1.5 percent of submissions from universities. The public debate focused on practices of gaming or ‘coding errors’ within university submissionsas the reason for this outcome. The issue was about the in/appropriate allocation of research activities to Fields of Research. This paper argues that such practices are only part of the explanation. With the support of statistical modelling, unrated outcomes are shown to have also arisen from particular evaluation practices within the discipline of Psychology and the associated Medical and Health Sciences Research Evaluation Committee. Given the high stakes nature of unrated outcomes and that the evaluation process breaches public administration principles by being not appealable nor appropriately transparent, the paper concludes with recommendations for the strengthening ERA policy and procedures to enhance trust in future ERA processes.  \nKeywords: higher education policy, universities, research performance, performance assessment, performance measurement, Excellence in Research for Australia, ERA, Australian Research Council, ARC, public administration  \nIntroduction  \nOn 4 December 2015, the Australian Research Council (ARC) released the results of its third Excellence in Research for Australia (ERA) assessment process. The process assessed disciplines, or Fields of Research (FoR), in each of 41 Australian universities, based on their own submissions. Each submission, or Unit of Evaluation (UoE), consisted of research indicators including research funding, research publications, applied indicators, and esteem measures. Each discipline at each university that made a submission was rated from 1 to 5, being‘well below world standard’ to being ‘well above world standard’. Where institutions did not make a submission  \nfor a particular discipline, a category of‘not assessed’(n/a) was given.  \nThe ERA process has been subjected to academic analysis and commentary (Gable, 2013), as has the wider and growing practice of national research assessment exercises, such as the UK’s Research Excellence Framework (REF) and its predecessor, the Research Assessment Exercise (RAE) (Elton, 2000; Geuna & Martin, 2003) . Many of these articles are highly critical of these forms of evaluation and their authors have argued that they do not achieve their stated goals, and also lead to dysfunctional managerial control over traditional academic practices (Bonnell, 2016; Kwok, 2013; Martin, 2011).While many of these articles challenge the performance metric  \n58 When rating systems do not rate Paul Henman, Scott D Brown & Simon Dennis vol. 59, no. 1, 2017  \nA U S T R A L I A N U N I V E R S I T I E S ’ R E V I E W  \napproach, many more engage with the contours and In an effort to better understand the administrative  \nspecifics of performance evaluation design. For example, black box, we examined the ERA production of n/r  \nthe ERA 2010 allocation of a four-scale ranking to journals UoE using both textual analysis of public reporting and  \nwas subjected to such widespread challenge due to its statistical examination of the outcomes. One of our goals  \nsubjective and political construction (Haslam & Koval, was to further investigate and understand the locus of 2010;Vanclay, 2011) that it was finally dropped. the problems associated with n/r outcomes. Furthermore,  \nIn ERA 2015 a new twist to the evaluation process our analyses provide a basis for suggestions for enhancing  \noccurred. In addition to the five-scale rating, a new the public governance of ERA processes to ensure they  \ncategory was created,‘not rated’(n/r).This new outcome maintain confidence and credib","cbCaiifcLB9OOdSx","https://ap.wps.com/l/cbCaiifcLB9OOdSx","pdf",580455,11,"English","# Introduction\n## ERA assessment design and rating categories\n## Background: critique of national research assessment\n## Emergence of “not rated” outcomes in ERA 2015\n## Research approach and objectives\n## Administrative opacity, confidentiality, and governance concerns","[{\"question\":\"What does the paper investigate about ERA 2015?\",\"answer\":\"The paper investigates why rating systems failed to rate a portion of university submissions in ERA 2015, focusing on both reported coding-error explanations and other evaluation practices.\"},{\"question\":\"How does ERA rate research indicators and what new outcome appeared in 2015?\",\"answer\":\"ERA rates disciplines from 1 to 5 and uses “not assessed” when no submission is made; in 2015 it also introduced “not rated” (n/r) for disciplinary areas where committees could not assign a numerical rating due to coding errors.\"},{\"question\":\"Why does the paper argue that unrated outcomes raise governance concerns?\",\"answer\":\"It notes that unrated outcomes are high-stakes and the evaluation process breaches public administration principles by being not appealable and lacking appropriate transparency.\"},{\"question\":\"What are the paper’s main recommendations?\",\"answer\":\"The paper concludes with recommendations aimed at strengthening ERA policy and procedures to enhance confidence and credibility in future ERA rounds.\"}]","A USTRALIAN UNIVERSITIES’ REVIEW - When rating systems do not rate - Evaluating ERA’s performance | PDF",1792079616,28]