The Multicollinearity Effect on the Performance of Machine Learning Algorithms - Case Examples in Healthcare Modelling

Aug 3, 2026

Upload by Skyler

Multicollinearity arises when exponentially expanding datasets contain highly correlated measurements, degrading both statistical and machine learning performance. Existing statistical remedies have not been evaluated sufficiently against modern learning approaches, motivating a comprehensive comparison. Regression models—Ridge, Liu, Lasso, and Elastic Net—are benchmarked alongside eight major machine learning methods using two multicollinearity-prone healthcare datasets (Body Fat, Cancer). Cross-validation evaluates RMSE, MAE, and R-squared, showing statistical models outperform machine learning across training and testing. Liu regression delivers particularly strong relative gains and remains underused despite promising results for regression-based studies under multicollinearity.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

INTRODUCTION

Multicollinearity problem in large-scale data

Differences between statistics and machine learning approaches

Study motivation and contributions

Research Article Academic Platform Journal of Engineering and Smart Systems (APJESS) 12(3), 68-80, 2024 Received: 04-Oct-2023 Accepted: 14-Aug-2024 homepage: [https://dergipark.org.tr/tr/pub/apjess](https://dergipark.org.tr/tr/pub/apjess) [https://doi.org/10.21541/apjess.1371070](https://doi.org/10.21541/apjess.1371070) The Multicollinearity Effect on the Performance of Machine Learning Algorithms: Case Examples in Healthcare Modelling *1 Hasan YILDIRIM *1 Corresponding Author, Department of Mathematics, Karamanoğlu Mehmetbey University, Türkiye, [hasanyildirim@kmu.edu.tr](hasanyildirim@kmu.edu.tr) Abstract The data extracted from various fields inherently consists of extremely correlated measurements in parallel with the exponential increase in the size of the data that need to be interpreted owing to the technological advances. This problem, called the multicollinearity, influences the performance of both statistical and machine learning algorithms. Statistical models proposed asa potential remedy to this problem have not been sufficiently evaluated in the literature. Therefore, a comprehensive comparison of statistical and machine learning models is required for addressing the multicollinearity problem. Statistical models (including Ridge, Liu, Lasso and Elastic Net regression) and the eight most important machine learning algorithms (including Cart, Knn, Mlp, MARS, Cubist, Svm, Bagging and XGBoost) are comprehensively compared by using two different healthcare datasets (including Body Fat and Cancer) having multicollinearity problem. The performance of the models is assessed through cross validation methods via root mean square error, mean absolute error and r-squared criteria. The results of the study revealed that statistical models outperformed machine learning models in terms of root mean square error, mean absolute error and r-squared criteria in both training and testing performance. Particularly the Liu regression often achieved better relative performance (up to 7.60% to 46.08% for Body Fat data set and up to 1.55% to 21.53% for Cancer data set on training performance and up to 1.56% to 38.08% for Body Fat data set and up to 3.50% to 23.29% for Cancer data set on testing performance) among regression methods as well as compared to machine algorithms. Liu regression is mostly disregarded in the machine learning literature, but since it outperforms the most powerful and widely used machine learning algorithms, it appears to be a promising tool in almost all fields, especially for regression-based studies including data with multicollinearity problem. Keywords: Machine learning; Multicollinearity; Feature selection; Collinearity; Artificial intelligence 1. INTRODUCTION Mathematical modeling mainly involves the processes of prediction and inference using a set of explanatory variables (i.e. attributes) that are considered to have an effect on a particular ([i.e. target](i.e. target)) variable. Facilitated by technological advances, the data collection process has significantly increased the scale of the variables. There have emerged highly correlated measurements that are assessed in almost every field, especially in areas such as health, marketing and finance [1] . In big databases containing thousands of variables, it is inevitable that complex patterns of relationships between variables will be discovered. The relationship is considered reasonable to a certain extent, but if it is extreme, a phenomenon known in the statistical literature as multicollinearity (i.e. collinearity) arises [2] . The multicollinearity problem stands out as a problem that is encountered quite frequently in the increasing data size with the ease of data collection in real life problems but is generally under-emphasized [3] . However, mathematically, this problem causes both statistical and machine learning models to often yield inaccurate inferences and poor predictions (i.e. generalization ability) . The appr

FAQs

What problem does the study focus on?

The study examines multicollinearity and how highly correlated measurements can reduce prediction accuracy and generalization ability in both statistical and machine learning models.

Which model types are compared in the experiments?

The research compares statistical regression models (Ridge, Liu, Lasso, Elastic Net) with eight machine learning algorithms (including CART, KNN, MLP, MARS, Cubist, SVM, Bagging, and XGBoost).

What evaluation method and metrics are used to assess performance?

Model performance is assessed using cross-validation with RMSE, MAE, and R-squared criteria on two healthcare datasets (Body Fat and Cancer).

Aug 3, 2026

Upload by Skyler