Explaining is Harder Than Predicting Alone Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

Jul 16, 2026

In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labeled examples, yet the use of provided context is unclear. This paper evaluates concept-based explainability for frozen MLLMs under few-shot ICL using five increasingly formal conditions, from baseline classification to Description Logics (DL) axiom generation, across four state-of-the-art models via an LLM-as-a-judge pipeline. Results show explaining is harder than predicting: forcing structured explanations monotonically reduces accuracy, while explanation quality correlates with correct predictions when discriminative visual features are articulated. The study concludes current MLLMs lack instruction-tuning for machine-verifiable formal explanations.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages25

Reading Time1 h 3 min

Introduction

Related Work

Methodology and Explanation Conditions

Evaluation Framework and Judge Model

Experimental Design

Experimental Results

Discussion

Conclusion

Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers arXiv :2605 .282 15v2 [ cs .AI] 10 Jul 2026 Carmen Quiles-Ramrez 1 Leticia L. Rodrguez 2 Abstract In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labelled examples. Yet how these models use the provided context remains opaque. While Chain-of-Thought prompting is widely used, recent work argues it may not reflect true internal computation (Barez et al., 2025) . In this paper, we systematically evaluate the conceptbased explainability of frozen MLLMs under fewshot ICL using five conditions of increasing formal rigour: from baseline classification to Description Logics (DL) axiom generation. Evaluating four state-of-the-art MLLMs via an independent LLM-as-a-judge pipeline, we demonstrate that explaining is genuinely harder than predicting alone. Surprisingly, forcing models to generate formally structured, concept-based explanations degrades predictive accuracy monotonically (from 93.8% to 90.1%), contradicting the assumption that explicit reasoning universally aids performance. However, when models successfully articulate class-discriminative visual features, explanation quality strongly correlates with correct predictions. Our findings suggest that while MLLMs excel at visual classification, they lack the specific instruction-tuning required for formal, machineverifiable explainability. 1. Introduction In-context learning (ICL) (Brown et al., 2020) has emerged as a powerful paradigm for adapting large language models to new tasks without parameter updates. Extended to the multimodal setting, MLLMs can now classify images from a few labelled examples provided directly in the context window (Dong et al., 2024) . Despite impressive few-shot 1Universidad de Granada, Spain 2Universidad de Buenos Aires, Argentina. Correspondence to: Carmen Quiles-Ram´ırez <car[menqr@ugr.es](menqr@ugr.es) >. Accepted to the 2nd Workshop on Compositional Learning atICML 2026, Seoul, South Korea. Copyright 2026 by the author(s) . Nicols Martorell 2 Natalia Daz-Rodrguez 1 accuracy, the reasoning process underlying these predictions remains a black box: models produce answers without any principled account of why a particular image belongs to a given class. The dominant approach to eliciting explanations from large models is chain-of-thought (CoT) prompting, which asks the model to verbalise intermediate reasoning steps. However, Barez et al. (2025) argue that CoT does not constitute genuine explainability: the generated text may diverge from the model’s actual internal computation, offering only a plausible-sounding post-hoc narrative. Similarly, Huang et al. (2025) show that VLMs tend to mimic rather than reason from ICL context, further questioning whether CoTstyle outputs constitute genuine understanding. This gap motivates the search for concept-based explanations (Poeta et al., 2025)—formal statements that are both humanreadable and machine-verifiable. We approach this problem through the lens of eXplainable AI (XAI) (Ali et al., 2023) and Description Logics (Baader et al., 2009), a decidable fragment of first-order logic widely used for knowledge representation. By designing a hierarchy of five explanation conditions of increasing formal complexity, we systematically investigate whether state-ofthe-art MLLMs can extract the discriminative visual features of an image class, formalise them as IF–THEN rules, and ultimately express them as DL. We evaluate explanation quality using an independent LLM-as-a-judge pipeline anda set of contributed XAI metrics. Studying explanations within few-shot image classification provides a highly controlled setting. It encourages the model to ground its responses in newly introduced classes, helping to mitigate the influence of broad pre-training biases. Our primary goal is to assess how MLLMs explain the knowledge derive

FAQs

What problem does the paper address about multimodal ICL image classification?

It studies why the reasoning behind image-classification answers is opaque, even though models can produce labels from a few examples in the context.

How does the paper evaluate concept-based explanations of MLLMs?

It systematically tests five explanation conditions with increasing formal rigor, from free-text explanations to generating Description Logics (DL) axioms, using an LLM-as-a-judge evaluation pipeline and multiple quality metrics.

What are the main findings about the relationship between explanations and prediction accuracy?

Generating formally structured, concept-based explanations degrades predictive accuracy monotonically, but when models successfully articulate class-discriminative visual features, explanation quality strongly correlates with correct predictions.

Jul 16, 2026

Upload by Olivia Brown