Removing the effects of the site in brain imaging machine-learning - Measurement and extendable benchmark

Aug 3, 2026

Multisite machine-learning neuroimaging studies must remove differences between acquisition sites to avoid effects of the site (EoS), which can bias or artificially inflate prediction-model performance. Prior approaches aiming to remove EoS (e.g., ComBat) may fail when EoS are complex, including interactions between brain regions, and complex EoS can further distort accuracy. The work proposes a strategy to measure how effectively a MAREoS removes distinct EoS types. FORMAREOS provides multisite MRI datasets and formulas to compute relative accuracy change, plus an extendable benchmark website for algorithm- and EoS-specific evaluation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Introduction

Background and motivation

Proposed measurement strategy

FORMAREOS datasets and formulas

Extendable benchmark website

Implications for different machine-learning algorithms

NeuroImage 265 (2023) 119800 Contents lists available at ScienceDirect NeuroImage journal [homepage: www.elsevier.com/locate/neuroimage](homepage: www.elsevier.com/locate/neuroimage) | Removing the eﬀects of the site in brain imaging machine-learning – Measurement and extendable benchmark | | | | | --- | --- | --- | --- | | Aleix Solanesa,b,+, Corentin J Goslingc,d,m,+, Lydia Forteaa,e,f, María Ortuñoa,
Elisabet Lopez-Soleya,f,g, Sara Llufriua,f,g, Santiago Maderoa,e,f,h, Eloy Martinez-Herasa,f,g, Edith Pomarol-Clotete,i,j, Elisabeth Solana a,f,g, Eduard Vietaa,e,f,h, Joaquim Raduaa,e,f,k,l,∗
a Institut d’Investigacions Biomèdiques August Pi i Sunyer (IDIBAPS), Barcelona, Spain
b Department of Psychiatry and Forensic Medicine, Autonomous University of Barcelona, Barcelona, Spain cDysCo Lab, Paris Nanterre University, Nanterre, France
d Laboratoire de Psychopathologie et Processus de Santé, Université de Paris, Paris, France
e Biomedical Network Research Centre on Mental Health (CIBERSAM), Instituto de Salud Carlos III, Madrid, Spain f University of Barcelona, Barcelona, Spain
g Center of Neuroimmunology, Laboratory of Advanced Imaging in Neuroimmunological Diseases, Hospital Clinic Barcelona h Barcelona Bipolar Disorders and Depressive Unit, Institute of Neurosciences, Hospital Clinic, Barcelona, Spain iFIDMAG Germanes Hospitalàries Research Foundation, Barcelona, Spain
j Benito Menni CASM, Sant Boi de Llobregat, Barcelona, Spain
k Department of Psychosis Studies, Institute of Psychiatry, Psychology, and Neuroscience, King’s College London, London, United Kingdom l Centre for Psychiatric Research and Education, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
m Centre for Innovation in Mental Health (CIMH), School of Psychology, Faculty of Environmental and Life Sciences, University of Southampton, Southampton, UK | | | | | a r t i c l e i n f o | | a b s t r a c t | | | Keywords:
Benchmark
Eﬀects of the site Machine-learning Magnetic resonance imaging | | Multisite machine-learning neuroimaging studies, such as those conducted by the ENIGMA Consortium, need to remove the diﬀerences between sites to avoid eﬀects of the site (EoS) that may prevent or fraudulently help the creation of prediction models, leading to impoverished or inﬂated prediction accuracy. Unfortunately, we have shown earlier that current Methods Aiming to Remove the EoS (MAREoS, e.g., ComBat) cannot remove complex EoS (e.g., including interactions between regions). And complex EoS may bias the accuracy. To overcome this hurdle, groups worldwide are developing novel MAREoS. However, we cannot assess their eﬀectiveness because EoS may either inﬂate or shrink the accuracy, and MAREoS may both remove the EoS and degrade the data. In this work, we propose a strategy to measure the eﬀectiveness of a MAREoS in removing diﬀerent types of EoS. FORMAREOS DEVELOPERS, we provide two multisite MRI datasets with only simple true eﬀects (i.e., detectable by most machine-learning algorithms) and two with only simple EoS (i.e., removable by most MAREoS). First, they should use these datasets to ﬁt machine-learning algorithms after applying the MAREoS. Second, they should use the formulas we provide to calculate the relative accuracy change associated with the MAREoS in each dataset and derive an EoS-removal eﬀectiveness statistic. We also oﬀer similar datasets and formulas for complex true eﬀects and EoS that include ﬁrst-order interactions. FOR MACHINE-LEARNING RESEARCHERS, we provide an extendable benchmark website to show: a) the types of EoS they should remove for each given machine-learning algorithm and b) the eﬀectiveness of each MAREoS for removing each type of EoS. Relevantly, a MAREoS only able to remove the simple EoS may suﬃce for simple machine-learning algorithms, whereas more complex algorithms need a MAREoS that can remove more complex EoS. For instance, ComBat removes all simple Eo

FAQs

Why is removing effects of the site (EoS) important in machine-learning neuroimaging?

Combining data across sites introduces site-related differences that can bias analyses. EoS can prevent or even falsely help the creation of prediction models by distorting prediction accuracy.

What limitation do earlier methods aiming to remove EoS (MAREoS) have?

Current MAREoS, such as ComBat, cannot fully remove complex EoS that involve interactions between brain regions. Complex EoS may therefore continue to bias accuracy after harmonization.

How does the proposed approach measure MAREoS effectiveness?

The strategy fits machine-learning algorithms on multisite MRI datasets after applying a given MAREoS, then uses provided formulas to compute relative accuracy change. This yields an EoS-removal effectiveness statistic for each EoS type.

Aug 3, 2026

Upload by Emma Wilson