Losing faith in the classification and evaluation of research - A meta-metrics approach to research on religion in Australia

Oct 15, 2026

Uploaded by Mary Man

Analyzes research quality assessment in Australia’s Religion and Religious Studies field using a meta-metrics method. Compares outcomes reported through the Australian Research Council’s Excellence in Research for Australia process with global ranking indicators, identifying an apparent decline in perceived quality. Argues the quality loss may stem from how Field of Research code 2204 is constructed, and highlights disparities between peer-review disciplines and citation-based evaluation approaches.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages7

Reading Time18 min

AccessFree to download

Introduction

Metrics and meta-metrics in research evaluation

Background: development of bibliometrics and citation analysis

Critiques of using metrics for research quality and impact

Surveillance, data collection, and governance (context section)

A U S T R A L I A N U N I V E R S I T I E S ’ R E V I E W Losing faith in the classification and evaluation of research A meta-metrics approach to research on religion in Australia Adam Possamai & Gary Long Western Sydney University This article uses a meta-metrics approach to research the research in Religion and Religious Studies (Field of Research (FoR) Code 2204) in Australia. Comparing and contrasting various results from the data provided by the Australian Research Council (ARC) on its Excellence in Research for Australia (ERA) process, as well as global rankings data, the findings point to a reported drop of quality of research in this field. In this article we argue that this loss of quality could be due to the way this FoR Code is constructed and we call for its revision. This research also points to a considerable disparity between peer review disciplines such as FoR Code 2204 and citation disciplines. Keywords: ERA, religion, metrics, citations, FoR Codes Introduction This article is a response to the call from Wilsdon et al.(2015) recommending more research on research. To enact this, it re-appropriates the metrics system in a meta-metric approach to shed light on the work that academics are undertaking in Australia in the field of religion and religious studies. As such, it is not aimed at comparing and measuring individuals and/or organisations, but at allowing academics in this field to understand the impact on their work of metrics, peer assessment, and processes, and, hopefully, give them an understanding of their alienation from the products of their labour. The aim of this research is to provide tangible evidence, for the first time, concerning the academic success of the broad field of religion and religious studies as categorised and assessed by official government channels in Australia. The field of study under research here is categorised as 2204 Religion and Religious Studies and is one of the 157 FoR codes used to classify research activities in Australia and New Zealand. It is a code that includes a topic of research (religion as researched by, for example, anthropologists and historians) and a specific discipline (religious studies including, in this classification, theology). The use of metrics for research is not a new process. Rennet al. (2016) trace it back to 350 years ago when bibliometrics were used to statistically analyse publications. The first citation analysis was used in 1927 to produce a ranking of chemistry journals, and when the impact factor of journals was created as an indicator in the 1960s by the Institute for Scientific Information, it was meant to provide librarians with information to help them decide which journals to subscribe to. It was only later that this metric led to the ranking of journals to evaluate the performance of researchers and institutions. However, this article is not a discussion on the validity and social use of the research metrics currently vol. 62, no. 1, 2020 Losing faith in the classification and evaluation of research Adam Possamai & Gary Long 3 A U S T R A L I A N U N I V E R S I T I E S ’ R E V I E W available. This has been done quite extensively elsewhere. Findings can be supportive of the status quo such as the work of Terämäet al. (2016) on the metrics used by the Research Excellence Framework (REF) system in the UK. They found that measurement of research quality and research impact from institutions are aligned, and that there is little evidence that the current pursuit of impact detracts from the quality of research. Other more critical research can vary from proposing the notion of responsible metrics to deal with the unintended consequences of this system (Wilsdonet al., 2015), to a policy critic (Woelert, 2015), to a Marxist observation that academics are losing control of their surplus time (Hall, 2018). There are other types of critiques ofthe way quality is measured; these are on how citations and journal ranki

FAQs

What is the core method used in the article?

The article applies a meta-metrics approach to study research in Religion and Religious Studies in Australia by combining and comparing different evaluation and ranking data sources.

Why does the article claim research quality may be declining?

It argues the decline could be linked to how Field of Research code 2204 is constructed and how research is categorized and evaluated within that framework.

How does the article relate peer review disciplines to citation disciplines?

It points to a disparity between peer review disciplines (such as FoR code 2204) and citation-based disciplines, implying they do not align cleanly in measuring quality and impact.

Oct 15, 2026

Uploaded by Mary Man