Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site - Effects of site in machine learning - Abstract and methods

Aug 4, 2026

Upload by Miles

Brain MRI researchers conducting multisite studies, including within the ENIGMA Consortium, actively control site effects (EoS) during statistical analyses. In contrast, machine-learning MRI studies may remove EoS only during training, while failing to account for EoS when estimating model accuracy, which can yield substantially biased performance metrics. Examples from toy simulations and real MRI data show that removing EoS from both training and test still inflates (or occasionally shrinks) accuracy unless EoS is controlled during accuracy estimation. Practical control methods and an R package (“multisite.accuracy”) support multiple metrics.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

AccessFree to download

Introduction

Site effects in multisite MRI

EoS control in statistical analysis

EoS handling in machine-learning accuracy estimation

Solanes et al – Effects of site in machine-learning Biased accuracy in multisite machine-learning studies due to incomplete removal of the effects of the site Aleix Solanes1,2, Pol Palau3,4, Lydia Fortea1,5,6, Raymond Salvador3,5, Laura González-Navarro7, Cristian Daniel Llach1,5,6,8, Marc Valentí1,5,6,8, Eduard Vieta1,5,6,8, Joaquim Radua1,5,9,10,* 1 Institut d'Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain 2 Department of Psychiatry and Legal Medicine, Autonomous University of Barcelona, Barcelona, Spain. 3 FIDMAG Research Foundation, Barcelona, Spain 4 CASM Benito Menni Granollers-Hospital General de Granollers, Barcelona, Spain 5 Biomedical Network Research Centre on Mental Health (CIBERSAM), Instituto de Salud Carlos III, Madrid, Spain 6 Institute of Neurosciences, University of Barcelona, Barcelona, Spain 7 Faculty of Biology, University of Barcelona, Barcelona, Spain 8 Barcelona Bipolar Disorders and Depressive Unit, Institute of Neurosciences, Hospital Clinic, Barcelona, Spain 9 Department of Psychosis Studies, Institute of Psychiatry, Psychology, and Neuroscience, King's College London, London, UK 10 Centre for Psychiatric Research and Education, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden Running title: Effects of site in machine learning Keywords: bias; effects of the site; machine learning; magnetic resonance imaging * Author for correspondence: Dr. Joaquim Radua Institut d'Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS) . C/ . Rosselló, 149, 08036 Barcelona, Spain Phone: +34 932275707. Email: [radua@clinic.cat](radua@clinic.cat) Solanes et al – Effects of site in machine-learning ABSTRACT Brain MRI researchers conducting multisite studies, such as within the ENIGMA Consortium, are very aware of the importance of controlling the effects of the site (EoS) in the statistical analysis. Conversely, authors of the novel machine-learning MRI studies may remove the EoS when training the machine-learning models but not control them when estimating the models' accuracy, potentially leading to severely biased estimates. We show examples from a toy simulation study and real MRI data in which we remove the EoS from both the "training set" and the "test set" during the training and application of the model. However, the accuracy is still inflated (or occasionally shrunk) unless we further control the EoS during the estimation of the accuracy. We also provide several methods for controlling the EoS during the estimation of the accuracy, and a simple R package ("multisite.accuracy") that smoothly does this task for several accuracy estimates (e.g., sensitivity/specificity, area under the curve, correlation, hazard ratio, etc.) . Solanes et al – Effects of site in machine-learning 1. Introduction When an individual undergoes MRI brain scanning in different MRI devices, the resulting images differ (Focke et al., 2011) . These effects of the site (EoS) are relevant because, in many analyses, they may play a critical confounding role. To put a simple example, we might consider a two-site study investigating the effects of a disorder's severity on gray matter volume. Suppose that the sites are imbalanced: one site has individuals with mainly mild forms of the disease, and the other site has individuals with severe forms of the illness. In that case, the observed differences in gray matter volume between individuals with severe and mild conditions in the overall study could indeed represent differences between the two MRI scanning devices. Fortunately, researchers are very aware of the importance of the site's potentially confounding effects and the necessity of controlling them when conducting statistical analyses with tools such as ComBat, which removes differences in mean and variance related to the use of different MRI devices or sites (Radua et al., 2020) . However, analysts do not always control the EoS

FAQs

What problem does the paper identify in multisite machine-learning MRI studies?

Accuracy estimates can become biased when site effects are removed during training but not controlled when estimating model performance.

What do the simulation and real-data examples show?

Even after removing EoS from both training and test sets, accuracy remains inflated (or sometimes shrunk) unless EoS is additionally controlled during the estimation step.

How does the paper address the issue in practice?

It provides methods to control EoS during accuracy estimation and a simple R package (“multisite.accuracy”) that supports several accuracy metrics.

Aug 4, 2026

Upload by Miles