The harms of class imbalance corrections for machine learning based prediction models - a simulation study

Aug 4, 2026

Upload by Noah

Risk prediction models increasingly support healthcare decision making, where reliable calibration of risk estimates is essential for clinical use. Development data are often class-imbalanced, so researchers frequently apply imbalance corrections to balance event vs non-event observations. The study evaluates how imbalance corrections affect calibration across multiple machine learning algorithms. Extensive Monte Carlo simulations compare out-of-sample predictive performance with and without correction under varying sample sizes, predictor counts, and event fractions, and findings are illustrated using MIMIC-III data. Models trained without correction consistently achieve equal or better calibration; correction induces miscalibration via risk overestimation and re-calibration may not fully fix it. Correcting class imbalance is therefore not always necessary and can be harmful for clinically reliable individualized risk estimation.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

AccessFree to download

Abstract

Introduction

Purpose of clinical prediction models

Class imbalance and common correction practices

Calibration and consequences of miscalibration

Prior evidence and the research gap

Study approach and scope

arXiv :2404 . 19494v1 [ stat .ME] 30 Apr 2024 The harms of class imbalance corrections for machine learning based prediction models: a simulation study. A Preprint Alex Carriero Julius Center for Health Sciences and Primary Care University Medical Center Utrecht Netherlands [a.j.carriero@umcutrecht.nl](a.j.carriero@umcutrecht.nl) Kim Luijken Julius Center for Health Sciences and Primary Care University Medical Center Utrecht Netherlands Anne de Hond Julius Center for Health Sciences and Primary Care University Medical Center Utrecht Netherlands Karel GM Moons Julius Center for Health Sciences and Primary Care University Medical Center Utrecht Netherlands Ben van Calster Department of Development and Regeneration KU, Leuven Belgium Maarten van Smeden Julius Center for Health Sciences and Primary Care University Medical Center Utrecht Netherlands May 1, 2024 Abstract Risk prediction models are increasingly used in healthcare to aid in clinical decision making. In most clinical contexts, model calibration (i.e., assessing the reliability of risk estimates) is critical. Data available for model development are often not perfectly balanced with respect to the modeled outcome (i.e., individuals with vs. without the event of interest are not equally represented in the data) . It is common for researchers to correct this class imbalance, yet, the effect of such imbalance corrections on the calibration of machine learning models is largely unknown. We studied the effect of imbalance corrections on model calibration for a variety of machine learning algorithms. Using extensive Monte Carlo simulations we A preprint-May 1, 2024 compared the out-of-sample predictive performance of models developed with an imbalance correction to those developed without a correction for class imbalance across different datagenerating scenarios (varying sample size, the number of predictors and event fraction) . Our findings were illustrated in a case study using MIMIC-III data. In all simulation scenarios, prediction models developed without a correction for class imbalance consistently had equal or better calibration performance than prediction models developed with a correction for class imbalance. The miscalibration introduced by correcting for class imbalance was characterized by an over-estimation of risk and was not always able to be corrected with re-calibration. Correcting for class imbalance is not always necessary and may even be harmful for clinical prediction models which aim to produce reliable risk estimates on an individual basis. Keywords Class Imbalance · Machine Learning · Calibration · Prediction Modeling 1 Introduction Risk prediction models are increasingly used in healthcare to aid in clinical decision making; for example, to help decide if a patient is a good candidate for surgery or to communicate a patient’s risk of disease [1, 2 , 3] . As such, the purpose of a clinical prediction model is often to estimate a patient’s risk of experiencing a particular event (e.g., successful surgery, disease) [4, 5] . Due to the rarity of many diseases, data available to train clinical prediction models often exhibit class imbalance i.e. , observations from patients with vs. without the event of interest are not equally represented in the data. In machine learning literature, imbalance correction methods are commonly applied to correct class imbalance by artificially creating data that are more or perfectly balanced [6, 7 , 8 , 9], although the benefit of such corrections for model performance is not always clear. An abundance of imbalance correction methods exist [7, 8 , 9], yet, information regarding the effect of these imbalance corrections on model calibration is sparse. Model calibration captures the accuracy of risk estimates, relating to the agreement between the estimated (predicted) and observed number of events [3] . In clinical applications where a patient’s predicted risk is the e

FAQs

Why is calibration critical for healthcare risk prediction models?

Calibration measures whether predicted risks match observed event frequencies. Poor calibration can systematically over- or under-estimate risk or produce predictions that are too extreme or too modest, leading to misguided clinical decisions and potentially false reassurance.

What does this study investigate about class imbalance corrections?

It examines how common imbalance correction methods influence the calibration of machine learning-based risk prediction models, focusing on whether corrected models maintain reliable individualized risk estimates.

What are the main findings from the simulations and the case study?

Across simulation scenarios, models developed without class imbalance correction consistently show equal or better calibration performance. When correction is applied, miscalibration is characterized by risk overestimation and re-calibration does not always repair it; results are further illustrated using MIMIC-III data.

Aug 4, 2026

Upload by Noah