Systematic review finds “spin” practices and poor reporting standards in studies on machine learning-based prediction models

Aug 4, 2026

Upload by Eliana

Objectives evaluate the presence and frequency of spin practices and poor reporting standards in studies developing and/or validating clinical prediction models using supervised machine learning techniques. A systematic search of PubMed from 01/2018 to 12/2019 identified diagnostic and prognostic prediction model studies without restrictions on data sources, outcomes, or clinical specialty. Results included 152 studies, with frequent incomplete performance reporting and limited external validation before clinical recommendation. Conclusions indicate spin and inadequate reporting also occur in machine learning prediction model research, supporting a tailored framework for detection.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Introduction

Study objectives

Search strategy and inclusion

Results: reporting and external validation

Conclusions and implications

Journal of Clinical Epidemiology 158 (2023) 99e110 REVIEW Systematic review ﬁnds ‘‘spin’’ practices and poor reporting standards in studies on machine learning-based prediction models Constanza L. Andaur Navarroa,b, *, Johanna A.A. Damena,b, Toshihiko Takadaa, Steven W.J. Nijmana, Paula Dhimanc,d, Jie Mac, Gary S. Collinsc,d, Ram Bajpaie, Richard D. Rileye, Karel G.M. Moonsa,b, Lotty Hoofta,b aJulius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands bCochrane Netherlands, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands cCenter for Statistics in Medicine, NDORMS, University of Oxford, Oxford, UK dNIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK eCentre for Prognosis Research, School of Medicine, Keele University, Keele, UK Accepted 28 March 2023; Published online 5 April 2023 Abstract Objectives: We evaluated the presence and frequency of spin practices and poor reporting standards in studies that developed and/or validated clinical prediction models using supervised machine learning techniques. Study Design and Setting: We systematically searched PubMed from 01/2018 to 12/2019 to identify diagnostic and prognostic prediction model studies using supervised machine learning. No restrictions were placed on data source, outcome, or clinical specialty. Results: We included 152 studies: 38% reported diagnostic models and 62% prognostic models. When reported, discrimination was described without precision estimates in 53/71 abstracts (74.6% [95% CI 63.4e83.3]) and 53/81 main texts (65.4% [95% CI 54.6e74.9]) . Of the 21 abstracts that recommended the model to be used in daily practice, 20 (95.2%[95% CI 77.3e99.8]) lacked any external validation of the developed models. Likewise, 74/133 (55.6%[95% CI 47.2e63.8]) studies made recommendations for clinical use in their main text without any external validation. Reporting guidelines were cited in 13/152 (8.6%[95% CI 5.1e14.1]) studies. Conclusion: Spin practices and poor reporting standards are also present in studies on prediction models using machine learning techniques. A tailored framework for the identiﬁcation of spin will enhance the sound reporting of prediction model studies. 􀀁 2023 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY license ([http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/)). Keywords: Diagnosis; Prognosis; Development; Validation; Misinterpretation; Overinterpretation; Overextrapolation; Spin Systematic review registration: PROSPERO, CRD42019161764 . Funding: There is no speciﬁc funding to disclosure for this study. GSC is funded by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC) and by Cancer Research UK program grant (C49297/A27294) . PD is funded by the NIHR Oxford BRC. RB is afﬁliated to the National Institute for Health and Care Research (NIHR) Applied Research Collaboration (ARC) West Midlands. None of the funding sources had a role in the design, conduct, analyses, or reporting of the study or in the decision to submit the manuscript for publication. The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. Competing interests: Authors declare no competing interests. Availability of data, code, and other materials: Data and analytical code is available upon reasonable request to corresponding author. Declaration of interests: The authors declare that they have no known competing ﬁnancial interests or personal relationships that could have appeared to inﬂuence the work reported in this paper. Registration and protocol: This review was registered in PROSPERO (CRD42019161764). The study protocol can be accessed in [https://doi](https://doi). org/10.1136/bmjopen-2020-038832 .

FAQs

What does the review evaluate about machine learning prediction model studies?

It evaluates the presence and frequency of spin practices and poor reporting standards in studies that develop and/or validate supervised machine learning-based clinical prediction models.

How were studies selected for this systematic review?

The review systematically searched PubMed from 01/2018 to 12/2019 to identify diagnostic and prognostic prediction model studies using supervised machine learning, with no restrictions on data source, outcome, or clinical specialty.

What were the main findings regarding reporting and external validation?

Among 152 included studies, performance was often described without precision estimates, and most recommendations for daily clinical use lacked any external validation of the developed models.

Aug 4, 2026

Upload by Eliana