Generalized Variable Importance Metric - An approach to identify important predictors from machine learning models

Aug 4, 2026

Interpreting black box machine learning methods presents a major challenge because many existing explanations are tied to specific data or model structures. This thesis defines the “Generalized Variable Importance Metric (GVIM)” to quantify predictor importance using black box models without relying on model-based parameters. GVIM is defined via the true conditional expectation function and applies to continuous or binary outcomes. A permutation-based estimator is proposed, and bias under correlated predictors is analyzed through bias-variance decomposition, leading to a conditional GVIM extension. The methods are applied to cognitive decline risk factors using the CLSA dataset, offering model-agnostic, causal interpretability for clinical and public health research.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages152

Reading Time6 h 23 min

AccessFree to download

Abstract

Generalized Variable Importance Metric (GVIM)

Permutation-based estimation and correlated predictors

Conditional GVIM (CGVIM) and bias-variance decomposition

Application to cognitive decline using CLSA

Generalized Variable Importance Metric: An approach to identify important predictors from machine learning models by Mohammad Kaviul Anam Khan A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Biostatistics, Dalla Lana School of Public Health University of Toronto © Copyright 2025 by Mohammad Kaviul Anam Khan Generalized Variable Importance Metric: An approach to identify important predictors from machine learning models Mohammad Kaviul Anam Khan Doctor of Philosophy Graduate Department of Biostatistics, Dalla Lana School of Public Health University of Toronto 2025 Abstract Interpreting black box machine learning methods posses a significant challenge, with existing approaches often being data and model specific. In this thesis, a “Generalized Variable Importance Metric (GVIM)” is defined to measure predictor importance utilizing black box methods without relying on model-based parameters. GVIM, which is defined for a predictor using the true conditional expectation function, assesses the predictor’s impact on a continuous or binary response. A permutation-based approach to estimate GVIM is proposed in this thesis, akin to those by Breiman (2001) and Fisher et al. (2019a) . However, black-box models underestimate GVIM when predictors are correlated. Through a bias-variance decomposition, the source of the bias is identified and its pattern in high correlation scenarios is demonstrated, suggesting ways to minimize it. The primary bias stems from black-box models’ limited ability to extrapolate to regions that have low probability because of the correlations. A conditional GVIM method (CGVIM) based on Strobl et al. (2008) is introduced, its bias-variance decomposition is derived, and its relationship with predictor correlations is shown. Both GVIM and CGVIM exhibited a quadratic relationship with the conditional average treatment effect (CATE) . Finally, I demonstrated the application of GVIM and CGVIM to investigate risk factors for cognitive decline using data from the Canadian Longitudinal Study on Aging (CLSA) dataset. The proposed method is model-agnostic and offers a causal interpretation, which is crucial for clinical and public health research. Understanding exposure-outcome relationships is vital in health science, where traditional models like regression are preferred for interpretability, but machine learning excels in prediction. GVIM and CGVIM, being model-agnostic, allow researchers to choose their preferred machine learning model without sacrificing prediction or inference capabilities. There is nothing more important to me than my family. This thesis is dedicated to my father, Mohammad Manirul Anam Khan; my mother, Hamida Anam Khan; my brother, Mohammad Muzahidul Anam Khan; my sister-in-law, Lisana Shahrin; my niece, Manha Shahrin Anam; and my nephew, Ahyan Shafraz Anam. Acknowledgements More than nine years ago, I embarked on my journey at the Dalla Lana School of Public Health, University of Toronto, to pursue an MSc in Biostatistics, followed by a PhD program at the same institution. Throughout these years, I have encountered concepts I never imagined existed. Oneof the most transformative was the idea of “variable importance”, introduced to me by my supervisor, Professor Rafal Kustra. This concept reshaped my understanding of statistics and applied statistics, opening up numerous avenues for future research. I will always be profoundly grateful to Professor Kustra for this. Additionally, he introduced me to machine learning methods, genetics, methylation, and other fields that were previously unfamiliar to me. I feel truly fortunate to have had him as my supervisor. He consistently made time to meet with me, both in-person and online, helping me overcome my limitations and elevate my work. His patience and kindness have been invaluable, and I thank him from the bottom of my heart. Next, I would like to exp

FAQs

What problem does the thesis address in black-box machine learning interpretation?

It addresses the difficulty of interpreting black-box machine learning methods, since many existing variable-importance approaches are data- or model-specific.

How is GVIM defined and what outcomes can it measure?

GVIM is defined for a predictor using the true conditional expectation function, and it assesses the predictor’s impact on continuous or binary responses.

Why do black-box models underestimate GVIM when predictors are correlated?

The thesis shows that correlated predictors introduce a bias source that follows a identifiable pattern via a bias-variance decomposition; the main bias arises from limited extrapolation ability to low-probability regions created by the correlations.

Where were GVIM and CGVIM applied, and what is the benefit claimed?

They were applied to study risk factors for cognitive decline using the CLSA dataset. The proposed methods are model-agnostic and provide a causal interpretation, supporting both prediction and inference in health research.

Aug 4, 2026

Upload by Theodora