The Harms of Class Imbalance Corrections for Machine Learning Based Prediction Models - A Simulation Study

Aug 4, 2026

Risk prediction models increasingly support healthcare decision-making, yet model calibration—how reliably predicted risks match observed outcomes—is often overlooked when training data are class-imbalanced. In many clinical settings, event prevalence is not evenly represented for patients with versus without the event. Researchers commonly apply class imbalance corrections, but their impact on calibration of machine learning prediction models remains unclear. This study evaluates calibration consequences across multiple machine learning algorithms and simulation scenarios, illustrated with MIMIC-III data.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Introduction

Risk prediction and calibration

Class imbalance and common correction practices

Methods

Monte Carlo simulation design

Algorithms and data-generating scenarios

Case study using MIMIC-III

Results

Calibration comparison across scenarios

Characterization of miscalibration and re-calibration

Conclusion

Statistics in Medicine RESEARCH ARTICLE OPEN ACCESS The Harms of Class Imbalance Corrections for Machine Learning Based Prediction Models: A Simulation Study Alex Carriero1 | Kim Luijken1 | Anne de Hond1 | Karel G. M. Moons1 | Ben van Calster2 | Maarten van Smeden1 1Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands | 2 KU Leuven, Leuven, Belgium Correspondence: Alex Carriero ([a.j.carriero@umcutrecht.nl](a.j.carriero@umcutrecht.nl)) Received: 25 April 2024 | Revised: 19 September 2024 | Accepted: 13 December 2024 Funding: The authors received no specific funding for this work. Keywords: calibration | class imbalance | machine learning | prediction modeling ABSTRACT Introduction: Risk prediction models are increasingly used in healthcare to aid in clinical decision-making. In most clinical contexts, model calibration (i.e., assessing the reliability of risk estimates) is critical. Data available for model development are often not perfectly balanced with the modeled outcome (i.e., individuals with vs. without the event of interest are not equally prevalent in the data) . It is common for researchers to correct for class imbalance, yet, the effect of such imbalance corrections on the calibration of machine learning models is largely unknown. Methods: We studied the effect of imbalance corrections on model calibration for a variety of machine learning algorithms. Using extensive Monte Carlo simulations we compared the out-of-sample predictive performance ofmodels developedwith an imbalance correction to those developed without a correction for class imbalance across different data-generating scenarios (varying sample size, the number of predictors, and event fraction) . Our findings were illustrated in a case study using MIMIC-III data. Results: In all simulation scenarios, prediction models developed without a correction for class imbalance consistently had equal or better calibration performance than prediction models developed with a correction for class imbalance. The miscalibration introduced by correcting for class imbalance was characterized by an over-estimation ofrisk and was not always able tobe corrected with re-calibration. Conclusion: Correcting for class imbalance is not always necessary and may even be harmful to clinical prediction models which aim to produce reliable risk estimates on an individual basis. 1 | Introduction Risk prediction models are increasingly used in healthcare to aid in clinical decision-making; for example, to help decide ifa patient is a good candidate for surgery or to communicate a patient’s risk of disease [1–3] . As such, the purpose of a clinical prediction model is often to estimate a patient’s risk of experiencing a particular event (e.g., successful surgery, disease)[4, 5] . Due to the rarity of many diseases, data available to train clinical prediction models often exhibit class imbalance, that is, observations from patients with vs. without the event of interest are not equally represented in the data. In machine learning literature, imbalance correction methods are commonly applied to correct class imbalance by artificially creating data that are more or perfectly balanced (artificially creating an event prevalence of 0.50) [6–9] . Although, the benefit of such corrections for problems where true event prevalence is less than 0.50 is not always clear [10, 11] . This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made. © 2025 The Author(s) . Statistics in Medicine published by John Wiley & Sons Ltd. Statistics in Medicine, 2025; 44:e10320 1 of 22 [https://doi.org/10.1002/sim.10320](https://doi.org/10.1002/sim.10320) An abundance of imbalance correction metho

FAQs

Why is calibration important for clinical risk prediction models?

Calibration reflects how closely predicted risks match observed event rates. Poor calibration can cause systematic over- or under-estimation, leading to misleading clinical decisions and inappropriate reassurance to patients.

How does the study evaluate the effect of class imbalance corrections?

The study uses extensive Monte Carlo simulations to compare out-of-sample predictive performance of models trained with imbalance correction versus without, across varying sample sizes, numbers of predictors, and event fractions.

What is the main finding about class imbalance corrections and calibration?

Across all simulation scenarios, models developed without class imbalance correction consistently show equal or better calibration than models developed with correction. The harm is characterized by over-estimation of risk and may not be resolved by re-calibration.

Aug 4, 2026

Upload by River Wang