All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models

Jul 16, 2026

Explaining machine-learning models is critical for decision-making and consumer trust, yet Explainable AI (XAI) is often treated as trading predictive accuracy for explanation quality. This work argues the trade-off is not inherent: when explanation and prediction are properly coupled, they become complementary, and self-explaining models can improve accuracy. Existing XAI fails due to invalidity under distribution shift, limitations of scalar attributions for nonlinear relations, and weak linkage between explanations and predictions. The Rashomon Explanation paradigm constructs faithful, prediction-guiding explanation sets, proving non-emptiness and bounding guided model performance. An LLM agent workflow iteratively aligns explanations with predictions, converges under explicit conditions, and improves accuracy and explanation quality across churn, clinical survival regression, and clickthrough prediction on large-scale Kuaishou logs, holding under distribution shift.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages54

Reading Time2 h 16 min

Introduction

Accuracy–Explainability Trade-off in XAI

Three Limitations of Existing XAI Methods

arXiv :2607 .09502v 1 [ cs .LG] 10 Jul 2026 All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models Pan Li Scheller College of Business, [Georgia Tech.](Georgia Tech. pan. li@scheller. gatech. edu)[ pan. li@scheller. gatech. edu](Georgia Tech. pan. li@scheller. gatech. edu) Abstract Explaining machine-learning models is increasingly important for decision-making and consumer trust, yet it is widely believed to come at a cost: existing Explainable AI (XAI) methods suffer from a persistent accuracy–explainability trade-off. We argue that this trade-off is not fundamental, but rather an artifact of treating explanation and prediction as two separate objectives; when the two are properly coupled, they become complementary, such that equipping a model to explain itself improves, rather than degrades, its predictive accuracy. Realizing this benefit, however, requires overcoming three limitations of existing XAI methods: (1) no single explanation remains valid under distribution shift; (2) scalar attributions, such as SHAP, cannot capture conditional or nonlinear relationships; and (3) explanations are disconnected from the predictions they inform. We therefore introduce the Rashomon Explanation paradigm for building a set of faithful, predictionguiding explanations rather than a single one, and we prove that this set is generally non-empty and that the fidelity of these explanations bounds the performance of the models they guide. To explore the set of Rashomon Explanation, motivated by the organizational theories of sensemaking and double-loop learning, we propose RashomonLLM, an Explanation–Prediction– Reflection LLM agentic workflow that generates the explanation set in natural language by iteratively aligning the explanation with the prediction. We prove that under explicit conditions, RashomonLLM converges and recovers the full Rashomon Explanation set. Empirically, across three applications of customer-churn classification, clinical survival regression, and industrial clickthrough prediction on large-scale logs from Kuaishou’s live-streaming platform, RashomonLLM significantly outperforms state-of-the-art prediction and XAI baselines on both accuracy and explanation quality, and surpasses strong specialized deep CTR models. Furthermore, ablation, robustness, and calibration analyses confirm that this accuracy gain is achieved by equipping the model to explain itself and is driven by explanation fidelity, and that it persists across distribution shifts, temporal splits, and random seeds. As a result, our framework enables organizations to advance business performance and lay the groundwork for consumer trust at the same time. Keywords: Explainable AI (XAI); Large Language Models; Rashomon Set; Agentic Workflow; Interpretable Machine Learning. 1 INTRODUCTION Explainable artificial intelligence (XAI) has become increasingly popular thanks to its benefits in establishing consumer trust (McKinsey & Company, 2022), enhancing accountability, model diagnostics, and usability (Senoner et al., 2022; Wang et al., 2022), even though black-box alternatives (such as Deep Learning or LLMs) may achieve better performance. An example is loan approvals, where financial institutions use a model to assess the risk of an individual defaulting on the payment and generate an “approve/deny” decision. In this case, interpretable models 1 (such as Logistic Regression or Decision Tree) remain competitive baselines and are widely used in credit scoring (Lessmann et al., 2015), since the cost of being wrong and not knowing why is far greater than the benefit of being slightly more right; in fact, the Equal Credit Opportunity Act (ECOA) requires creditors to provide a specific reason if they take an adverse action against a consumer, such as denying their loan application (Barocas & Selbst, 2016), and it is obligatory for creditors to disclose these specific reasons to ensure fa

FAQs

What does the paper claim about the accuracy–explainability trade-off in XAI?

The paper argues the trade-off is not fundamental. By coupling explanation and prediction appropriately, explanations can improve predictive accuracy rather than degrade it.

Why are existing XAI explanations considered unreliable in this work?

The paper identifies three issues: explanations can stop being valid under distribution shift, scalar attributions like SHAP miss conditional or nonlinear relationships, and explanations are disconnected from the predictions they are meant to support.

What is the Rashomon Explanation paradigm, and how does RashomonLLM use it?

The Rashomon Explanation paradigm builds a set of faithful, prediction-guiding explanations instead of a single one. RashomonLLM uses an Explanation–Prediction–Reflection LLM agentic workflow that iteratively aligns explanations with predictions and is shown to converge under explicit conditions.

Jul 16, 2026

Upload by Theodora