[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-122912-en":3,"doc-seo-122912-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},122912,137441390410,"Hazel","https://ap-avatar.wpscdn.com/avatar/2000252f4ab5702993?_k=1776741390130283984",8,"Research & Report","Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research - version 2 action","Cost-effective integration of Microsoft Academic Graph (MAG) with adjunctive machine learning is evaluated for maintaining and updating living systematic reviews and maps in coronavirus disease 2019 (COVID-19) research. The study tests whether MAG serves as a sufficiently sensitive single source and whether eligible records can be retrieved with acceptable specificity. An eight-arm cost-effectiveness analysis compares semi-automated MAG-enabled workflows with conventional Boolean database searching. Results show improved recall and coverage alongside lower weekly costs in MAG-enabled approaches.","This is a repository copy of Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research.  \nWhite Rose Research Online URL for this paper:  \n[https://eprints.whiterose.ac.uk/215118/](https://eprints.whiterose.ac.uk/215118/)  \nVersion: Published Version  \nArticle:  \nShemilt, Ian, Arno, Anneliese, Thomas, James et al. (8 more authors) (2024) Costeffectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research. Wellcome Open Research. 210. ISSN 2398-502X  \n[https://doi.org/10.12688/wellcomeopenres.17141.2](https://doi.org/10.12688/wellcomeopenres.17141.2)  \nReuse  \nThis article is distributed under the terms of the Creative Commons Attribution (CC BY) licence. This licence allows you to distribute, remix, tweak, and build upon the work, even commercially, as long as you credit the authors for the original work. More information and the full terms of the licence here: [https://creativecommons.org/licenses/](https://creativecommons.org/licenses/)  \nTakedown  \nIf you consider content in White Rose Research Online to be in breach of UK law, please notify us by  \nemailing [eprints@whiterose.ac.uk](eprints@whiterose.ac.uk) including the URL of the record and the reason for the withdrawal request.  \n[eprints@whiterose.ac.uk](eprints@whiterose.ac.uk)[ ](eprints@whiterose.ac.uk)[https://eprints.whiterose.ac.uk/](https://eprints.whiterose.ac.uk/)  \nRESEARCH ARTICLE  \nCost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research [version 2; peer review: 2 approved]  \nIan Shemilt 1*, Anneliese Arno 1*, James Thomas 1*, Theo Lorenc2,  \nClaire Khouja 2, Gary Raine2, Katy Sutcliffe 1, D'Souza Preethy 1, Irene Kwan 1, Kath Wright2, Amanda Sowden2  \n1 EPPI-Centre, UCL Social Research Institute, University College London, London, London, WC1H 0NR, UK  \n2Centre for Reviews and Dissemination, University of York, UK, York, Yorkshire, UK  \n* Equal contributors  \nv2  \nFirst published: 19 Aug 2021, 6:210  \n[https://doi.org/10.12688/wellcomeopenres.17141.1](https://doi.org/10.12688/wellcomeopenres.17141.1)  \n[Latest published:](Latest published: 26 Mar 2024)[ 26 Mar 2024](Latest published: 26 Mar 2024), 6:210  \n[https://doi.org/10.12688/wellcomeopenres.17141.2](https://doi.org/10.12688/wellcomeopenres.17141.2)  \nAbstract  \nBackground  \nIdentifying new, eligible studies for integration into living systematic reviews and maps usually relies on conventional Boolean updating searches of multiple databases and manual processing of the updated results. Automated searches of one, comprehensive, continuously updated source, with adjunctive machine learning, could enable more efficient searching, selection and prioritisation workflows for updating (living) reviews and maps, though research is needed to establish this. Microsoft Academic Graph (MAG) is a potentially comprehensive single source which also contains metadata that can be used in machine learning to help efficiently identify eligible studies. This study sought to establish whether: (a) MAG was a sufficiently sensitive single source to maintain our living map of COVID-19 research; and (b) eligible records could be identified with an acceptably high level of specificity.  \nMethods  \nWe conducted an eight-arm cost-effectiveness analysis to assess the  \nOpen Peer Review  \n\n| Approval Status  |  |  |\n| --- | --- | --- |\n| 1 |  | 2 |\n| version 2\u003Cbr>(revision) | \u003Cbr>view | \u003Cbr>view |\n| 26 Mar 2024 |  |  |\n| version 1\u003Cbr>19 Aug 2021 | \u003Cbr>view | \u003Cbr>view |\n\n1. Irma Klerings , Danube University Krems, Krems, Austria  \n2. Su Golder , University of York, York, UK Any reports and responses or comments on the article can be found at the end of the article.  \ncosts, recall and precision of semi-automated workf","cbCaijqDmpWaIvXz","https://ap.wps.com/l/cbCaijqDmpWaIvXz","pdf",2836565,1,27,"English","en",105,"# Abstract\n## Background\n## Methods\n## Results\n## Conclusions\n# Keywords","[{\"question\":\"What problem does the study address in living systematic reviews and maps?\",\"answer\":\"Living reviews and maps require efficient identification of newly eligible studies. Conventional Boolean updating and manual processing across multiple databases are resource intensive, so automated approaches are needed.\"},{\"question\":\"How does the study evaluate cost-effectiveness and performance?\",\"answer\":\"It uses an eight-arm cost-effectiveness analysis comparing semi-automated MAG-enabled workflows with adjunctive machine learning against conventional Boolean searching methods, measuring recall, precision, and screening workload.\"},{\"question\":\"What were the main findings for MAG-enabled workflows with machine learning?\",\"answer\":\"The MAG-enabled workflow dominated conventional workflows in base case and sensitivity analyses. After one month, it identified 469 additional eligible articles and reduced costs by £3,179 per week versus conventional methods.\"}]","Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research - version 2 action | PDF",1785813623,68,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"cost-effectiveness-of-microsoft-academic-graph-with-machine-learning-for-automated-study-identification-in-a-living-map-of-coronavirus-disease-2019-covid-19-research-version-2-action","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/cost-effectiveness-of-microsoft-academic-graph-with-machine-learning-for-automated-study-identification-in-a-living-map-of-coronavirus-disease-2019-covid-19-research-version-2-action/122912/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the study address in living systematic reviews and maps?","Question",{"text":75,"@type":76},"Living reviews and maps require efficient identification of newly eligible studies. Conventional Boolean updating and manual processing across multiple databases are resource intensive, so automated approaches are needed.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the study evaluate cost-effectiveness and performance?",{"text":80,"@type":76},"It uses an eight-arm cost-effectiveness analysis comparing semi-automated MAG-enabled workflows with adjunctive machine learning against conventional Boolean searching methods, measuring recall, precision, and screening workload.",{"name":82,"@type":73,"acceptedAnswer":83},"What were the main findings for MAG-enabled workflows with machine learning?",{"text":84,"@type":76},"The MAG-enabled workflow dominated conventional workflows in base case and sensitivity analyses. After one month, it identified 469 additional eligible articles and reduced costs by £3,179 per week versus conventional methods.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]