[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117873-en":3,"doc-seo-117873-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},117873,34359740700684,"Finn","https://ap-avatar.wpscdn.com/avatar/1f400023980c374ae676?_k=1777273430885731487",8,"Research & Report","Exploring the application of machine learning to expert evaluation of research impact","This study investigates the application of machine learning techniques to large-scale human expert evaluation of the impact of academic research. Publicly available impact case study data from the UK Research Excellence Framework (2014) are used to train five machine learning models on qualitative and quantitative features, including institutional, disciplinary, narrative-style, bibliometric, and policy indicators. Results indicate that machine learning can make decisions resembling expert evaluators, revealing strong effects of institutional context and narrative metrics, while supporting a shift toward predictive analysis with caution for automated assessment.","King’s Research Portal  \nDOI:  \n10.1371/journal.pone.0288469  \nDocument Version  \nPublisher's PDF, also known as Version of record  \nLink to publication record in King's Research Portal  \nCitation for published version (APA):  \nWilliams, K. , Michalska, S. , Cohen, E. , Szomszor, M. , & Grant, J. (2023) . Exploring the application of machine learning to expert evaluation of research impact. PLoS One, 18(8 August),[e0288469] .  \n[https://doi.org/10.1371/journal.pone.0288469](https://doi.org/10.1371/journal.pone.0288469)  \nCiting this paper  \nPlease note that where the full-text provided on King's Research Portal is the Author Accepted Manuscript or Post-Print version this may differ from the final Published version. If citing, it is advised that you check and use the publisher's definitive version for pagination, volume/issue, and date of publication details. And where the final published version is provided on the Research Portal, if citing you are again advised to check the publisher's website for any subsequent corrections.  \nGeneral rights  \nCopyright and moral rights for the publications made accessible in the Research Portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognize and abide by the legal requirements associated with these rights.  \n•Users may download and print one copy of any publication from the Research Portal for the purpose of private study or research.  \n•You may not further distribute the material or use it for any profit-making activity or commercial gain  \n•You may freely distribute the URL identifying the publication in the Research Portal  \nTake down policy  \nIf you believe that this document breaches copyright please contact [librarypure@kcl.ac.uk](librarypure@kcl.ac.uk) providing details, and we will remove access to the work immediately and investigate your claim.  \nDownload date: 08. Oct. 2023  \nPLOS ONE  \nOPEN ACCESS  \nCitation: Williams K, Michalska S, Cohen E, Szomszor M, Grant J (2023) Exploring the application of machine learning to expert evaluation of research impact. PLoS ONE 18(8): e0288469 .  \n[https://doi.org/10.1371/journal.pone.0288469](https://doi.org/10.1371/journal.pone.0288469)  \n[Editor:](Editor: Steve Zimmerman)[ Steve Zimmerman](Editor: Steve Zimmerman), Public Library of  \nScience, UNITED KINGDOM Received: October 5, 2022  \nAccepted: June 27, 2023  \nPublished: August 3, 2023  \nCopyright: © 2023 Williams et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.  \nData Availability Statement: [https://ref.ac.uk/](https://ref.ac.uk/)[ ](https://ref.ac.uk/)[2014/api.openalex.org](2014/api.openalex.org) [https://www.overton.io](https://www.overton.io).  \nFunding: KW & JG. ES/V004123/1 UK Economic and Social Research Council (ESRC) [https://www](https://www). [ukri.org/councils/esrc/](ukri.org/councils/esrc/ The funder)[ The funder](ukri.org/councils/esrc/ The funder) had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.  \nCompeting interests: The authors have declared that no competing interests exist.  \nRESEARCH ARTICLE  \nExploring the application of machine learning to expert evaluation of research impact  \nKate Williams1 *, Sandra Michalska2, Eliel Cohen2, Martin Szomszor3, Jonathan Grant4  \n1 School of Social and Political Sciences, University of Melbourne, Melbourne, Victoria, Australia, 2 Policy Institute, King’s College London, London, Greater London, United Kingdom, 3 Electric Data Solutions, London, Greater London, United Kingdom, 4 Different Angles, Cambridge, Cambridgeshire, United Kingdom  \n* [kate.williams@unimelb.edu.au](kate.williams@unimelb.edu.au)  \nAbstract  \nThe objective of this study is to investigate the application of machine learning technique","cbCaimcxkRgngsWn","https://ap.wps.com/l/cbCaimcxkRgngsWn","pdf",1797642,1,19,"English","en",105,"# Abstract\n# Introduction","[{\"question\":\"What is the objective of the study?\",\"answer\":\"To investigate how machine learning can be applied to large-scale human expert evaluation of academic research impact.\"},{\"question\":\"Which data and features are used to train the models?\",\"answer\":\"The study uses publicly available impact case study data from the UK Research Excellence Framework (2014) and trains models on qualitative and quantitative features, including institution, discipline, narrative style, bibliometric indicators, and policy indicators.\"},{\"question\":\"What do the experiment results suggest about machine learning and expert evaluation?\",\"answer\":\"The results show that machine learning models can process information to make decisions resembling those of expert evaluators, with strong influence from institutional context and narrative-style metrics, though caution is needed for automated assessment.\"}]","Exploring the application of machine learning to expert evaluation of research impact | PDF",1785680093,48,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"exploring-the-application-of-machine-learning-to-expert-evaluation-of-research-impact","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/exploring-the-application-of-machine-learning-to-expert-evaluation-of-research-impact/117873/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the objective of the study?","Question",{"text":75,"@type":76},"To investigate how machine learning can be applied to large-scale human expert evaluation of academic research impact.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Which data and features are used to train the models?",{"text":80,"@type":76},"The study uses publicly available impact case study data from the UK Research Excellence Framework (2014) and trains models on qualitative and quantitative features, including institution, discipline, narrative style, bibliometric indicators, and policy indicators.",{"name":82,"@type":73,"acceptedAnswer":83},"What do the experiment results suggest about machine learning and expert evaluation?",{"text":84,"@type":76},"The results show that machine learning models can process information to make decisions resembling those of expert evaluators, with strong influence from institutional context and narrative-style metrics, though caution is needed for automated assessment.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},"General","general"]