Screening for in vitro systematic reviews - a comparison of screening methods and training of a machine learning classifier

Aug 5, 2026

Upload by Jiven

Objective: Strategies for identifying relevant studies for systematic review can vary in performance across research domains. This work compares four approaches that use either human or automated screening of title and abstract or full text, and describes training of a machine learning classifier to detect in vitro studies from bibliographic records. Methods: A systematic review of oxygen–glucose deprivation in PC-12 cells compared human screening and text mining using enriched PubMed Central training decisions. Results: Full-text methods performed best, while title/abstract screening with text mining outperformed human screening; the target sensitivity yielded moderate specificity. Conclusion: Title/abstract exclusions can be substantial; the algorithm can support first-phase selection to reduce full-text workload.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

AccessFree to download

Objective

Methods

Human screening

Automated screening and model training

Results

Conclusion

Introduction

Edinburgh Research Explorer Screening for in vitro systematic reviews Citation for published version: Wilson, E, Cruz, F, Maclean, D, Ghanawi, J, McCann, SK, Brennan, PM, Liao, J, Sena, ES & Macleod, M 2023, 'Screening for in vitro systematic reviews: a comparison of screening methods and training of a machine learning classifier', Clinical science, vol. 137, no. 2, pp. 181-193. [https://doi.org/10.1042/CS20220594](https://doi.org/10.1042/CS20220594) Digital Object Identifier (DOI): 10.1042/CS20220594 Link: Link to publication record in Edinburgh Research Explorer Document Version: Publisher's PDF, also known as Version of record Published In: Clinical science General rights Copyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s) and / or other copyright owners and it is a condition of accessing these publications that users recognise and abide by the legal requirements associated with these rights. Take down policy The University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorer content complies with UK legislation. If you believe that the public display of this file breaches copyright please [contact openaccess@ed.ac.uk](contact openaccess@ed.ac.uk) providing details, and we will remove access to the work immediately and investigate your claim. Download date: 11. May. 2024 Clinical Science (2023) 137 181–193 [https://doi.org/10.1042/CS20220594](https://doi.org/10.1042/CS20220594) Research Article Screening for in vitro systematic reviews: a comparison of screening methods and training of a machine learning classiﬁer Emma Wilson1 , Florenz Cruz2 , Duncan Maclean3 , Joly Ghanawi4 , Sarah K. McCann2 , Paul M. Brennan1 , Jing Liao1 , Emily S. Sena1 and Malcolm Macleod1 1 Centre for Clinical Brain Sciences, The University of Edinburgh, Edinburgh, U. K. ; 2 Berlin Institute of Health at Charit-Universittsmedizin Berlin, QUEST Center, Berlin, Germany; 3 University of Edinburgh Medical School, University of Edinburgh, Edinburgh, U. K. ; 4 Independent Researcher, U. K. Correspondence: Emma Wilson ([emma.wilson@ed.ac.uk](emma.wilson@ed.ac.uk)) Received: 31 August 2022 Revised: 15 December 2022 Accepted: 11 January 2023 Accepted Manuscript online: 11 January 2023 Version of Record published: 27 January 2023 Objective: Existing strategies to identify relevant studies for systematic review may not perform equally well across research domains. We compare four approaches based on either human or automated screening of either title and abstract or full text, and report the training of a machine learning algorithm to identify in vitro studies from bibliographic records. Methods: We used a systematic review of oxygen–glucose deprivation (OGD) in PC-12 cells to compare approaches. For human screening, two reviewers independently screened studies based on title and abstract or full text, with disagreements reconciled by a third. For automated screening, we applied text mining to either title and abstract or full text. We trained a machine learning algorithm with decisions from 2000 randomly selected PubMed Central records enriched with a dataset of known in vitro studies. Results: Full-text approaches performed best, with human (sensitivity: 0.990, specificity: 1 .000 and precision: 0.994) outperforming text mining (sensitivity: 0.972, specificity: 0.980 and precision: 0.764) . For title and abstract, text mining (sensitivity: 0.890, specificity: 0.995 and precision: 0.922) outperformed human screening (sensitivity: 0.862, specificity: 0.998 and precision: 0.975) . At our target sensitivity of 95% the algorithm performed with specificity of 0.850 and precision of 0.700. Conclusion: In this in vitro systematic review, human screening based on title and abstract erroneously excluded 14% of relevant studies, perhaps because title and abstract provide an incomplete description of methods used

FAQs

What is the primary objective of this study?

To compare multiple strategies for screening studies for in vitro systematic reviews, including human versus automated approaches and title/abstract versus full-text screening, and to report training of a machine learning classifier.

How were human and automated screening approaches implemented?

Human screening used two independent reviewers screening title/abstract or full text, with disagreements reconciled by a third reviewer. Automated screening applied text mining to title/abstract or full text.

Which screening approach performed best and what key trade-off was observed?

Full-text approaches performed best overall, with human screening achieving very high sensitivity and specificity. When targeting sensitivity at 95%, specificity and precision were lower, showing a trade-off between recall and exclusion accuracy.

Aug 5, 2026

Upload by Jiven