Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research - version 2 action

Aug 4, 2026

Upload by Hazel

Cost-effective integration of Microsoft Academic Graph (MAG) with adjunctive machine learning is evaluated for maintaining and updating living systematic reviews and maps in coronavirus disease 2019 (COVID-19) research. The study tests whether MAG serves as a sufficiently sensitive single source and whether eligible records can be retrieved with acceptable specificity. An eight-arm cost-effectiveness analysis compares semi-automated MAG-enabled workflows with conventional Boolean database searching. Results show improved recall and coverage alongside lower weekly costs in MAG-enabled approaches.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages27

Reading Time1 h 8 min

AccessFree to download

Abstract

Background

Methods

Results

Conclusions

Keywords

This is a repository copy of Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research. White Rose Research Online URL for this paper: [https://eprints.whiterose.ac.uk/215118/](https://eprints.whiterose.ac.uk/215118/) Version: Published Version Article: Shemilt, Ian, Arno, Anneliese, Thomas, James et al. (8 more authors) (2024) Costeffectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research. Wellcome Open Research. 210. ISSN 2398-502X [https://doi.org/10.12688/wellcomeopenres.17141.2](https://doi.org/10.12688/wellcomeopenres.17141.2) Reuse This article is distributed under the terms of the Creative Commons Attribution (CC BY) licence. This licence allows you to distribute, remix, tweak, and build upon the work, even commercially, as long as you credit the authors for the original work. More information and the full terms of the licence here: [https://creativecommons.org/licenses/](https://creativecommons.org/licenses/) Takedown If you consider content in White Rose Research Online to be in breach of UK law, please notify us by emailing [eprints@whiterose.ac.uk](eprints@whiterose.ac.uk) including the URL of the record and the reason for the withdrawal request. [eprints@whiterose.ac.uk](eprints@whiterose.ac.uk)[ ](eprints@whiterose.ac.uk)[https://eprints.whiterose.ac.uk/](https://eprints.whiterose.ac.uk/) RESEARCH ARTICLE Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research [version 2; peer review: 2 approved] Ian Shemilt 1*, Anneliese Arno 1*, James Thomas 1*, Theo Lorenc2, Claire Khouja 2, Gary Raine2, Katy Sutcliffe 1, D'Souza Preethy 1, Irene Kwan 1, Kath Wright2, Amanda Sowden2 1 EPPI-Centre, UCL Social Research Institute, University College London, London, London, WC1H 0NR, UK 2Centre for Reviews and Dissemination, University of York, UK, York, Yorkshire, UK * Equal contributors v2 First published: 19 Aug 2021, 6:210 [https://doi.org/10.12688/wellcomeopenres.17141.1](https://doi.org/10.12688/wellcomeopenres.17141.1) [Latest published:](Latest published: 26 Mar 2024)[ 26 Mar 2024](Latest published: 26 Mar 2024), 6:210 [https://doi.org/10.12688/wellcomeopenres.17141.2](https://doi.org/10.12688/wellcomeopenres.17141.2) Abstract Background Identifying new, eligible studies for integration into living systematic reviews and maps usually relies on conventional Boolean updating searches of multiple databases and manual processing of the updated results. Automated searches of one, comprehensive, continuously updated source, with adjunctive machine learning, could enable more efficient searching, selection and prioritisation workflows for updating (living) reviews and maps, though research is needed to establish this. Microsoft Academic Graph (MAG) is a potentially comprehensive single source which also contains metadata that can be used in machine learning to help efficiently identify eligible studies. This study sought to establish whether: (a) MAG was a sufficiently sensitive single source to maintain our living map of COVID-19 research; and (b) eligible records could be identified with an acceptably high level of specificity. Methods We conducted an eight-arm cost-effectiveness analysis to assess the Open Peer Review | Approval Status | | | | --- | --- | --- | | 1 | | 2 | | version 2
(revision) |
view |
view | | 26 Mar 2024 | | | | version 1
19 Aug 2021 |
view |
view | 1. Irma Klerings , Danube University Krems, Krems, Austria 2. Su Golder , University of York, York, UK Any reports and responses or comments on the article can be found at the end of the article. costs, recall and precision of semi-automated workf

FAQs

What problem does the study address in living systematic reviews and maps?

Living reviews and maps require efficient identification of newly eligible studies. Conventional Boolean updating and manual processing across multiple databases are resource intensive, so automated approaches are needed.

How does the study evaluate cost-effectiveness and performance?

It uses an eight-arm cost-effectiveness analysis comparing semi-automated MAG-enabled workflows with adjunctive machine learning against conventional Boolean searching methods, measuring recall, precision, and screening workload.

What were the main findings for MAG-enabled workflows with machine learning?

The MAG-enabled workflow dominated conventional workflows in base case and sensitivity analyses. After one month, it identified 469 additional eligible articles and reduced costs by £3,179 per week versus conventional methods.

Aug 4, 2026

Upload by Hazel