ALICE - Combining Feature Selection and Inter-Rater Agreeability for Machine Learning Insights

Aug 5, 2026

Upload by Liam

This paper introduces ALICE, a Python library that combines conventional feature selection with inter-rater agreeability to produce actionable insights for black-box machine learning models. The framework is developed from an overview of key interpretability concepts in ML, then its architecture, intuition, and main methods are discussed in detail. Initial experiments on a customer churn predictive modeling task report early results and suggest directions for future work. Source code and experiment notebooks are provided online.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

AccessFree to download

Introduction

Background and Related Work

Causal ML

Explainable AI

Inter-Rater Agreeability

Feature Selection

Proposed Framework and Experiments

ALICE: COMBINING FEATURE SELECTION AND INTER-RATER AGREEABILITY FOR MACHINE LEARNING INSIGHTS arXiv :2404 .09053v1 [ cs .LG] 13 Apr 2024 Bachana Anasashvili Humboldt University of Berlin [bachana.anasashvili@student.hu-berlin.de](bachana.anasashvili@student.hu-berlin.de) Vahidin Jeleskovic Humboldt University of Berlin [vahidin.jeleskovic@hu-berlin.de](vahidin.jeleskovic@hu-berlin.de) ABSTRACT This paper presents a new Python library called Automated Learning for Insightful Comparison and Evaluation (ALICE), which merges conventional feature selection and the concept of inter-rater agreeability in a simple, user-friendly manner to seek insights into black box Machine Learning models. The framework is proposed following an overview of the key concepts of interpretability in ML. The entire architecture and intuition of the main methods of the framework are also thoroughly discussed and results from initial experiments on a customer churn predictive modeling task arepresented, alongside ideas for possible avenues to explore for the future. The full source code for the framework and the experiment notebooks can be found at: [https://github.com/anasashb/aliceHU](https://github.com/anasashb/aliceHU). 1 Introduction The use of Machine Learning models for decision-making has become the new norm not only in tech but any business field imaginable, covering any possible task at hand be it search engine recommendations, customer churn prediction, credit risk scoring, energy load forecasting, or the deployment of personalized AI assistants. This comes at a time when developing ML models has become increasingly easier with the rise of open-source, free and user-friendly Python libraries such as Keras, scikit-learn, PyTorch and as generative AI-based conversational chatbots such as ChatGPT, Gemini and Claude that can provide coding assistance—if not ready-made code for modeling—are evolving rapidly. Such developments yet again beg the question of interpretability in machine learning, which has been formulated in various ways in literature and been offered multiple proposed solutions such as exploring causality (see Section 2.1), explainability (see Section 2.2) or abandoning black box ML models altogether. But to make a philosophical argument, it is hard to see the benefits of highly model or domain-specific, post-hoc, or complex solutions to obtain insights into the inner-doings of machine learning models when the modeling task itself is growing ever more accessible to laypeople. Common thought on categorizing ML models in this regard would argue that parametric models descending from the fields of statistics and econometrics such as Linear or Logistic Regression are by nature more interpretable than their data-driven and non-parametric counterparts such as tree-based models or neural networks. While we do not in any form challenge this widely-held assumption, we inquire if there’s a possibility to gauge the degree of differences between the outcomes from these different models. There are troves of literature that compare a vast array of learning algorithms on this or that predictive modeling task offering analysis based on error metrics solely. Yes, when deploying a model in a business, the best performance is what may matter most—after all, the smaller the error the better. Let us assume an overly simplified scenario where an ML engineer is handed data with a sizeable amount of features and tasked to build a predictive model. We would not be incorrect to assume that in its most skeletal form, the following process would include some sort of feature selection, evaluation runs with different model architectures, and a final decision based on which model performs best on the validation set. While knowing that a hypothetical Random Forest performed better on the hypothetical task than a hypothetical Logistic Regression or a Multi-Layer Perceptron may be sufficient enough, what we suggest is that it could also be be

FAQs

What is ALICE and what problem does it address?

ALICE is a Python library that merges feature selection with inter-rater agreeability. It targets the need for additional insight into black-box machine learning model behavior beyond raw performance metrics.

How does the paper frame ML interpretability?

It reviews interpretability through multiple lenses discussed in the literature, including causality, explainability, and inter-rater agreeability. The motivation is to support understanding even when modeling tasks are increasingly accessible.

What kind of experiment results are reported?

The paper presents initial experimental results using a customer churn predictive modeling task. It also outlines possible future directions to extend the framework.

Aug 5, 2026

Upload by Liam