Exploring the application of machine learning to expert evaluation of research impact

Aug 2, 2026

Upload by Finn

This study investigates the application of machine learning techniques to large-scale human expert evaluation of the impact of academic research. Publicly available impact case study data from the UK Research Excellence Framework (2014) are used to train five machine learning models on qualitative and quantitative features, including institutional, disciplinary, narrative-style, bibliometric, and policy indicators. Results indicate that machine learning can make decisions resembling expert evaluators, revealing strong effects of institutional context and narrative metrics, while supporting a shift toward predictive analysis with caution for automated assessment.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Abstract

Introduction

King’s Research Portal DOI: 10.1371/journal.pone.0288469 Document Version Publisher's PDF, also known as Version of record Link to publication record in King's Research Portal Citation for published version (APA): Williams, K. , Michalska, S. , Cohen, E. , Szomszor, M. , & Grant, J. (2023) . Exploring the application of machine learning to expert evaluation of research impact. PLoS One, 18(8 August),[e0288469] . [https://doi.org/10.1371/journal.pone.0288469](https://doi.org/10.1371/journal.pone.0288469) Citing this paper Please note that where the full-text provided on King's Research Portal is the Author Accepted Manuscript or Post-Print version this may differ from the final Published version. If citing, it is advised that you check and use the publisher's definitive version for pagination, volume/issue, and date of publication details. And where the final published version is provided on the Research Portal, if citing you are again advised to check the publisher's website for any subsequent corrections. General rights Copyright and moral rights for the publications made accessible in the Research Portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognize and abide by the legal requirements associated with these rights. •Users may download and print one copy of any publication from the Research Portal for the purpose of private study or research. •You may not further distribute the material or use it for any profit-making activity or commercial gain •You may freely distribute the URL identifying the publication in the Research Portal Take down policy If you believe that this document breaches copyright please contact [librarypure@kcl.ac.uk](librarypure@kcl.ac.uk) providing details, and we will remove access to the work immediately and investigate your claim. Download date: 08. Oct. 2023 PLOS ONE OPEN ACCESS Citation: Williams K, Michalska S, Cohen E, Szomszor M, Grant J (2023) Exploring the application of machine learning to expert evaluation of research impact. PLoS ONE 18(8): e0288469 . [https://doi.org/10.1371/journal.pone.0288469](https://doi.org/10.1371/journal.pone.0288469) [Editor:](Editor: Steve Zimmerman)[ Steve Zimmerman](Editor: Steve Zimmerman), Public Library of Science, UNITED KINGDOM Received: October 5, 2022 Accepted: June 27, 2023 Published: August 3, 2023 Copyright: © 2023 Williams et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: [https://ref.ac.uk/](https://ref.ac.uk/)[ ](https://ref.ac.uk/)[2014/api.openalex.org](2014/api.openalex.org) [https://www.overton.io](https://www.overton.io). Funding: KW & JG. ES/V004123/1 UK Economic and Social Research Council (ESRC) [https://www](https://www). [ukri.org/councils/esrc/](ukri.org/councils/esrc/ The funder)[ The funder](ukri.org/councils/esrc/ The funder) had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist. RESEARCH ARTICLE Exploring the application of machine learning to expert evaluation of research impact Kate Williams1 *, Sandra Michalska2, Eliel Cohen2, Martin Szomszor3, Jonathan Grant4 1 School of Social and Political Sciences, University of Melbourne, Melbourne, Victoria, Australia, 2 Policy Institute, King’s College London, London, Greater London, United Kingdom, 3 Electric Data Solutions, London, Greater London, United Kingdom, 4 Different Angles, Cambridge, Cambridgeshire, United Kingdom * [kate.williams@unimelb.edu.au](kate.williams@unimelb.edu.au) Abstract The objective of this study is to investigate the application of machine learning technique

FAQs

What is the objective of the study?

To investigate how machine learning can be applied to large-scale human expert evaluation of academic research impact.

Which data and features are used to train the models?

The study uses publicly available impact case study data from the UK Research Excellence Framework (2014) and trains models on qualitative and quantitative features, including institution, discipline, narrative style, bibliometric indicators, and policy indicators.

What do the experiment results suggest about machine learning and expert evaluation?

The results show that machine learning models can process information to make decisions resembling those of expert evaluators, with strong influence from institutional context and narrative-style metrics, though caution is needed for automated assessment.

Aug 2, 2026

Upload by Finn