Comparing machine learning algorithms by union-free generic depth

Aug 4, 2026

We propose a framework to descriptively analyze samples of partial orders using depth functions. While depth has been intensively studied for linear and metric spaces, it remains largely unexplored for non-standard data types such as partial orders. The paper introduces the union-free generic (ufg) depth via an adaptation of simplicial depth on the space of all partial orders. Using ufg depth with multidimensional performance measures, it enables classifier comparisons on standard benchmark datasets, highlighting differences from existing benchmarking approaches.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

AccessFree to download

Abstract

Keywords

1. Introduction and related literature

1.1. Motivation

International Journal of Approximate Reasoning 169 (2024) 109166 Contents lists available at ScienceDirect International Journal of Approximate Reasoning journal [homepage: www.elsevier.com/locate/ijar](homepage: www.elsevier.com/locate/ijar) | Comparing machine learning algorithms by union-free generic depth | | | | | --- | --- | --- | --- | | Hannah Blocher ∗ , Georg Schollmeyer, Malte Nalenz, Christoph Jansen Department of Statistics, LMU Munich, Ludwigstr. 33, Munich, 80539, Bavaria, Germany | | | | | A R T I C L E I N F O | | A B S T R A C T | | | Keywords:
Partial orders
Data depth Benchmarking Algorithm comparison Outlier detection Non-standard data | | We propose a framework for descriptively analyzing sets of partial orders based on the concept of depth functions. Despite intensive studies in linear and metric spaces, there is very little discussion on depth functions for non-standard data types such as partial orders. We introduce an adaptation of the well-known simplicial depth to the set of all partial orders, the union-free generic (ufg) depth. Moreover, we utilize our ufg depth for a comparison of machine learning algorithms based on multidimensional performance measures. Concretely, we provide two examples of classiﬁer comparisons on samples of standard benchmark data sets. Our results demonstrate promisingly the wide variety of diﬀerent analysis approaches based on ufg methods. Furthermore, the examples outline that our approach diﬀers substantially from existing benchmarking approaches, and thus adds a new perspective to the vivid debate on classiﬁer comparison.1 | | 1. Introduction and related literature We begin with the general motivation for this paper and an overview of the contributions of our paper to the comparison of machine learning algorithms. We also provide references to related literature. 1.1. Motivation Partial orders – and the systematic incomparabilities of objects encoded in them – occur naturally in a variety of problems in a wide range of scientiﬁc disciplines. Examples range from decision theory, where the agents under consideration might be unable to arrange the consequences of their actions into total orders (see, e.g., [52,35]) or have partial cardinal preferences (see, e.g., [31,29]), over social choice theory, where a fair aggregate order might only be possible by incorporating systematic incomparabilities (see, e.g., [43,32]), to ﬁnance, where risky assets do not always have to be comparable (see, e.g., [37,13]). Of course, many other relevant examples exist. In the speciﬁc context of statistics and machine learning, the incompleteness of the considered orders often originates from the fact that the objects to be ordered are compared with respect to several criteria and/or on several instances simultaneously: only if there is unanimous dominance of one object over another, this order is included in the corresponding relation. Quite a number of * Corresponding author. [E-mail address:](E-mail address: hannah.blocher@stat.uni-muenchen.de)[ hannah.blocher@stat.uni-muenchen.de](E-mail address: hannah.blocher@stat.uni-muenchen.de) (H. Blocher). 1 Open Science: Reproducible implementation and data analysis are available at: [https://github.com/hannahblo/Comparing_Algorithms_Using_UFG_Depth](https://github.com/hannahblo/Comparing_Algorithms_Using_UFG_Depth). [https://doi.org/10.1016/j.ijar.2024.109166](https://doi.org/10.1016/j.ijar.2024.109166) Received 8 December 2023; Received in revised form 14 February 2024; Accepted 4 March 2024 Available online 7 March 2024 0888-613X/© 2024 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC license ([http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)). H. Blocher, G. Schollmeyer, M. Nalenz et al. International Journal of Approximate Reasoning 169 (2024) 109166 research papers recently have been devoted to such compar

FAQs

What problem does the paper address for classifier comparison?

It addresses how to compare classifiers when performance is represented through partial orders with systematic incomparabilities rather than a single total ranking.

What is the proposed union-free generic (ufg) depth?

It is an adaptation of simplicial depth to the set of all partial orders, providing a depth function tailored to poset-valued data.

How is ufg depth used in the paper?

The paper applies ufg depth to compare machine learning algorithms using multidimensional performance measures, illustrated through two examples on standard benchmark datasets.

Aug 4, 2026

Upload by Theodore