A USTRALIAN UNIVERSITIES’ REVIEW - When rating systems do not rate - Evaluating ERA’s performance

Oct 15, 2026

Upload by Ben

This paper analyzes the Australian Government’s Excellence in Research for Australia (ERA) 2015 outcomes in which rating systems did not rate 1.5% of university research submissions. It examines public claims about gaming or “coding errors” and argues unrated results also stem from evaluation practices within Psychology and the Medical and Health Sciences Research Evaluation Committee. It highlights governance failures—non-appealability and limited transparency—and closes with recommendations to strengthen ERA policy and procedures to build future trust.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Introduction

ERA assessment design and rating categories

Background: critique of national research assessment

Emergence of “not rated” outcomes in ERA 2015

Research approach and objectives

Administrative opacity, confidentiality, and governance concerns

A U S T R A L I A N U N I V E R S I T I E S ’ R E V I E W When rating systems do not rate Evaluating ERA’s performance Paul Henman University of Queensland Scott D Brown & Simon Dennis University of Newcastle In 2015, the Australian Government’s Excellence in Research for Australia (ERA) assessment of research quality declined to rate 1.5 percent of submissions from universities. The public debate focused on practices of gaming or ‘coding errors’ within university submissionsas the reason for this outcome. The issue was about the in/appropriate allocation of research activities to Fields of Research. This paper argues that such practices are only part of the explanation. With the support of statistical modelling, unrated outcomes are shown to have also arisen from particular evaluation practices within the discipline of Psychology and the associated Medical and Health Sciences Research Evaluation Committee. Given the high stakes nature of unrated outcomes and that the evaluation process breaches public administration principles by being not appealable nor appropriately transparent, the paper concludes with recommendations for the strengthening ERA policy and procedures to enhance trust in future ERA processes. Keywords: higher education policy, universities, research performance, performance assessment, performance measurement, Excellence in Research for Australia, ERA, Australian Research Council, ARC, public administration Introduction On 4 December 2015, the Australian Research Council (ARC) released the results of its third Excellence in Research for Australia (ERA) assessment process. The process assessed disciplines, or Fields of Research (FoR), in each of 41 Australian universities, based on their own submissions. Each submission, or Unit of Evaluation (UoE), consisted of research indicators including research funding, research publications, applied indicators, and esteem measures. Each discipline at each university that made a submission was rated from 1 to 5, being‘well below world standard’ to being ‘well above world standard’. Where institutions did not make a submission for a particular discipline, a category of‘not assessed’(n/a) was given. The ERA process has been subjected to academic analysis and commentary (Gable, 2013), as has the wider and growing practice of national research assessment exercises, such as the UK’s Research Excellence Framework (REF) and its predecessor, the Research Assessment Exercise (RAE) (Elton, 2000; Geuna & Martin, 2003) . Many of these articles are highly critical of these forms of evaluation and their authors have argued that they do not achieve their stated goals, and also lead to dysfunctional managerial control over traditional academic practices (Bonnell, 2016; Kwok, 2013; Martin, 2011).While many of these articles challenge the performance metric 58 When rating systems do not rate Paul Henman, Scott D Brown & Simon Dennis vol. 59, no. 1, 2017 A U S T R A L I A N U N I V E R S I T I E S ’ R E V I E W approach, many more engage with the contours and In an effort to better understand the administrative specifics of performance evaluation design. For example, black box, we examined the ERA production of n/r the ERA 2010 allocation of a four-scale ranking to journals UoE using both textual analysis of public reporting and was subjected to such widespread challenge due to its statistical examination of the outcomes. One of our goals subjective and political construction (Haslam & Koval, was to further investigate and understand the locus of 2010;Vanclay, 2011) that it was finally dropped. the problems associated with n/r outcomes. Furthermore, In ERA 2015 a new twist to the evaluation process our analyses provide a basis for suggestions for enhancing occurred. In addition to the five-scale rating, a new the public governance of ERA processes to ensure they category was created,‘not rated’(n/r).This new outcome maintain confidence and credib

FAQs

What does the paper investigate about ERA 2015?

The paper investigates why rating systems failed to rate a portion of university submissions in ERA 2015, focusing on both reported coding-error explanations and other evaluation practices.

How does ERA rate research indicators and what new outcome appeared in 2015?

ERA rates disciplines from 1 to 5 and uses “not assessed” when no submission is made; in 2015 it also introduced “not rated” (n/r) for disciplinary areas where committees could not assign a numerical rating due to coding errors.

Why does the paper argue that unrated outcomes raise governance concerns?

It notes that unrated outcomes are high-stakes and the evaluation process breaches public administration principles by being not appealable and lacking appropriate transparency.

What are the paper’s main recommendations?

The paper concludes with recommendations aimed at strengthening ERA policy and procedures to enhance confidence and credibility in future ERA rounds.

Oct 15, 2026

Upload by Ben