Comprehensive Evaluation of Machine Learning Experiments - Algorithm Comparison, Algorithm Performance and Inferential Reproducibility

Aug 3, 2026

Upload by Eliana

This doctoral thesis examines key methodological issues in machine learning experimentation to improve how algorithm performance is evaluated, analyzed, and interpreted. It critiques the widely used train-dev-test paradigm by highlighting neglected sources of uncertainty, including algorithm variability and dependence on meta-parameters. The work develops a comprehensive statistical framework based on Linear Mixed Effects Models to model expected risk probabilistically, quantify reliability and performance homogeneity, and connect annotation processes to algorithm effects. It further proposes a unified approach to support inferential reproducibility across datasets.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages140

Reading Time5 h 53 min

AccessFree to download

Introduction

Research Question and Contribution

Algorithm Comparison

The Principles of Statistical Hypothesis Testing

Model-Based Algorithm Comparison: Toolbox

Model Based Algorithm Comparison: Analyzing an Example

Algorithm Performance Analysis

Untangling Terminology: Reliability, Agreement etc

A Unifying View on Algorithm Performance and Annotation

State of the Art Methods for Annotation and Algorithm Performance Analysis

Bridging the Gap: Model-based Reliability Analysis

Inferential Reproducibility

A Scheme for Analyzing Inferential Reproducibility

Linear Mixed Effects Models

Generalized Likelihood Ratio Tests

Variance Component Analysis and Reliability Coefficie

Comprehensive Evaluation of Machine Learning Experiments Algorithm Comparison, Algorithm Performance and Inferential Reproducibility Inauguraldissertation zur Erlangung des akademischen Grades Doktor der Philosophie der Neuphilologischen Fakultät der Ruprecht-Karls-Universität Heidelberg vorgelegt von Michael Hagmann am 07. August 2023 Hierbei handelt es sich um eine Heidelberger Dissertation. Erstgutachter: Prof. Dr. Stefan Riezler Institut für Computerlinguistik, Universität Heidelberg Zweitgutachter: Prof. Dr. Katja Markert Institut für Computerlinguistik, Universität Heidelberg Abstract This doctoral thesis addresses critical methodological aspects within machine learning experimentation, focusing on enhancing the evaluation and analysis of algorithm performance. The established "train-dev-test paradigm" commonly guides machine learning practitioners, involving nested optimization processes to optimize model parameters and meta-parameters and benchmarking against test data. However, this paradigm overlooks crucial aspects, such as algorithm variability and the intricate relationship between algorithm performance and meta-parameters. This work introduces a comprehensive framework that employs statistical techniques to bridge these gaps, advancing the methodological standards in empirical machine learning research. The foundational premise of this thesis lies in differentiating between algorithms and classifiers, recognizing that an algorithm may yield multiple classifiers due to inherent stochasticity or design choices. Consequently, algorithm performance becomes inherently probabilistic and cannot be captured by a single metric. The contributions of this work are structured around three core themes: Algorithm Comparison: A fundamental aim of empirical machine learning research is algorithm comparison. To this end, the thesis proposes utilizing Linear Mixed Effects Models (LMEMs) for analyzing evaluation data. LMEMs offer distinct advantages by accommodating complex data structures beyond the typical independent and identically distributed (iid) assumption. Thus LMEMs enable a holistic analysis of algorithm instances and facilitate the construction of nuanced conditional models of expected risk, supporting algorithm comparisons based on diverse data properties. Algorithm Performance Analysis: Contemporary evaluation practices often treat algorithms and classifiers as black boxes, hindering insights into their performance and parameter dependencies. Leveraging LMEMs, specifically implementing Variance Component Analysis, the thesis introduces methods from psychometrics to quantify algorithm performance homogeneity (reliability) and assess the influence of meta-parameters on performance. The flexibility of LMEMs allows a granular analysis of this relationship and extends these techniques to analyze data annotation processes linked to algorithm performance. Inferential Reproducibility: Building upon the preceding chapters, this section showcases a unified approach to analyze machine learning experiments comprehensively. By leveraging the full range of generated model instances, the analysis provides a nuanced understanding of competing algorithms. The outcomes offer implementation guidelines for algorithmic modifications and consolidate incongruent findings across diverse datasets, contributing to a coherent empirical perspective on algorithmic effects. This work underscores the significance of addressing algorithmic variability, metaparameter impact, and the probabilistic nature of algorithm performance. This thesis aims to enhance machine learning experiments’ transparency, reproducibility, and interpretability by introducing robust statistical methodologies facilitating extensive empirical analysis. It extends beyond conventional guidelines, offering a principled approach to advance the understanding and evaluation of algorithms in the evolving landscape of machine learning and data scien

FAQs

How does the thesis address limitations of the train-dev-test paradigm?

It identifies that the paradigm overlooks algorithm variability and the complex relationship between algorithm performance and meta-parameters. The thesis introduces statistical methods to model these effects more comprehensively.

What statistical approach is proposed for algorithm comparison?

The thesis proposes using Linear Mixed Effects Models (LMEMs) to analyze evaluation data while accommodating complex data structures. This enables nuanced conditional models of expected risk for comparisons under different data properties.

How is inferential reproducibility supported in the thesis?

It presents a unified analysis strategy that leverages the full range of generated model instances. The approach consolidates findings across datasets and provides guidance for principled algorithm modifications.

Aug 3, 2026

Upload by Eliana