An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learning

Aug 4, 2026

Upload by Rowan

The Rashomon Effect captures how, for a given dataset, multiple models can achieve equally strong performance while relying on different solution strategies. This paper examines consequences for Explainable Machine Learning, focusing on the comparability of explanations. A unified framework is proposed across three comparison scenarios, and a quantitative study evaluates diverse datasets, model families, attribution methods, and similarity metrics. Findings show hyperparameter tuning influences explanation outcomes and that metric choice substantially affects evaluation reliability.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Introduction

Rashomon Effect and XML context

Comparing Attribution Scores

Attribution score dependencies

Experimental setup and scenarios

Same model, same sample, and attribution method

Hyperparameter tuning and metric selection

Results and discussion

Perspective-specific findings

Conclusion

arXiv :2306 . 15786v2 [ cs .LG] 29 Jun 2023 An Empirical Evaluation of the Rashomon Effect in Explainable Machine Learning Sebastian M¨uller (􀀀)1 ,4[0000−0002−0778−9695], Vanessa Toborek 1 ,4[0009−0009−8372−8251], Katharina Beckh3 ,4[0000−0002−7824−6647], Matthias Jakobs2 ,4[0000−0003−4607−8957], Christian Bauckhage 1 ,3 ,4[0000−0001−6615−2128], and Pascal Welke5[0000−0002−2123−3781] 1 University of Bonn, Bonn, Germany 2 TU Dortmund University, Dortmund, Germany 3 Fraunhofer IAIS, Sankt Augustin, Germany 4 Lamarr Institute, Germany 5 TU Wien, Vienna, Austria Abstract. The Rashomon Effect describes the following phenomenon: for a given dataset there may exist many models with equally good performance but with different solution strategies. The Rashomon Effect has implications for Explainable Machine Learning, especially for the comparability of explanations. We provide a unified view on three different comparison scenarios and conduct a quantitative evaluation across different datasets, models, attribution methods, and metrics. We find that hyperparameter-tuning plays a role and that metric selection matters. Our results provide empirical support for previously anecdotal evidence and exhibit challenges for both scientists and practitioners. Keywords: Explainable ML · Interpretable ML · Attribution Methods · Rashomon Effect · Disagreement Problem 1 Introduction We demonstrate the impact of the Rashomon Effect when analyzing ML models. The Rashomon Effect [8] describes the phenomenon that there may exist many models within a hypothesis class which solve a dataset equally well. The set of these models is referred to as the Rashomon Set [12, 37] . From a data-centric perspective this phenomenon is also called Predictive Multiplicity [23], meaning that there exist many strategies to solve a task on a dataset. Other works use Rashomon Sets to analyze and describe data [12,30] . Somewhat surprisingly, the Rashomon Effect has not yet found wider attention in the Explainable Machine Learning (XML) literature. Although a few works have observed the effect it was only anecdotally or without referring to its proper name [14,20,35] . Accepted for presentation at European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD 2023) 2 M¨uller et al. XML has recently become a very active area of research and numerous explanation methods exist [1, 9, 24] . Many approaches explain black-box models in a post-hoc manner by providing attribution scores [22,27] which assign each input dimension a numerical value that represents this feature’s importance with respect to the model decision. Attribution scores are used to answer questions such as “What feature was the most important in this input sample?” and have been used to uncover spurious correlations in the data [29] and biased behavior of models [21] . However, attribution scores are sometimes ambiguous and their interpretation depends on the application context. It is hard to decide at what magnitude a feature is still important, particularly, if magnitudes of attribution scores can be sorted into an evenly descending order. It follows that the task of comparing different attribution methods is a difficult problem. Several works touch upon the problem of explanation comparison [5,7,19,26,35] from different perspectives. Our main contribution is an empirical analysis of one novel and two existing perspectives, 1) demonstrating model-specific sensitivity regarding the hyperparameter choice for explanation methods, 2) comparison of different explanations from the same attribution method on differently initialized but otherwise identical model architectures [5,35] and 3) the disagreement between different explanations applied to the same architecture and parameterization [19,26] . We place these three perspectives into a unified framework to investigate how the Rashomon Effect manifests itself in each situation.

FAQs

What is the Rashomon Effect in Explainable Machine Learning?

For a fixed dataset, many models can reach comparable performance but use different solution strategies. This creates a Rashomon Set of equally good yet behaviorally diverse models, which complicates how explanations should be compared.

Why is comparing attribution methods difficult according to the paper?

Attribution scores can be ambiguous and their interpretation depends on application context and on how magnitudes relate to importance. This makes it hard to determine whether two explanations agree or are meaningfully different.

What factors most affect explanation comparisons in the study?

The evaluation finds that hyperparameter tuning for explanation methods plays an important role. It also shows that the selection of similarity/assessment metrics can change conclusions about explanation quality or disagreement.

Aug 4, 2026

Upload by Rowan