When Do Explanations Help In-Context Learning? - A Comparative Study of Natural Language Explanation Types and Faithfulness

Sep 15, 2026

Uploaded by Aurora

Natural language explanations (NLEs) are increasingly used to influence model behavior in in-context learning (ICL), yet their relative effectiveness across explanation types remains unclear. This study compares four instruction-tuned models across six benchmarks, analyzing effects of NLE source (human, self-generated, external LLM) and selection strategy (random vs faithfulness-based). Results show frequent gains on classification-style tasks from adding NLEs, with external LLM explanations performing strongly and self-generated explanations depending more on selection. On math reasoning, outcomes vary by model and source, and faithfulness metrics may conflict, yielding different selected examples and downstream utility.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages53

Reading Time2 h 14 min

AccessFree to download

Abstract

Introduction

Motivation and background

NLE roles in real workflows

Research gap and objectives

arXiv :2608 . 16627v1 [ cs .CL] 17 Aug 2026 When Do Explanations Help In-Context Learning? A Comparative Study of Natural Language Explanation Types and Faithfulness Mahdi Dhaini∗ MaiNLP lab, CIS, LMU Munich Munich Center for Machine Learning (MCML) Adam Dejl Department of Computing, Imperial College London Juraj Vladika Technical University of Munich Volkan Özer Technical University of Munich, Munich Center for Machine Learning (MCML) Barbara Plank MaiNLP lab, CIS, LMU Munich Munich Center for Machine Learning (MCML) Gjergji Kasneci Technical University of Munich, Munich Center for Machine Learning (MCML) [mahdi. dhaini@lmu. de](mahdi. dhaini@lmu. de) [adam.dejl18@imperial. ac.uk](adam.dejl18@imperial. ac.uk) [juraj.vladika@tum. de](juraj.vladika@tum. de) [volkan. oezer@tum. de](volkan. oezer@tum. de) [b.plank@lmu. de](b.plank@lmu. de) [gjergji.kasneci@tum. de](gjergji.kasneci@tum. de) Abstract Natural language explanations (NLEs) are increasingly used as inputs, for example, as fewshot rationales that influence model behavior in in-context learning (ICL) . However, it remains unclear how different types of NLEs compare in their effects on downstream model performance in explanation-augmented prompting. Therefore, we provide a comparative evaluation across six benchmarks and four instruction-tuned models, studying how NLE source (human-written when available, self-generated explanations, generated by an external LLM) and NLE selection (random vs faithfulness-based filtering) affect downstream utility of NLEs when used in ICL settings. Our extensive evaluation shows that, on classificationstyle benchmarks, adding NLEs to few-shot prompts often improves accuracy over few-shot prompting without explanations; among NLE sources, externally generated LLM-NLEs often provide strong downstream utility and remain competitive with human rationales where both are available, whereas self-NLEs are more sensitive to the selection strategy. On math reasoning, the effects are more model-and source-dependent. We further show that faithfulness-based selection of self-NLEs yields small average gains overall, but can improve or reduce performance depending on the metric, task, and model. Different faithfulness metrics can disagree substantially, affecting which self-NLE examples are selected and their downstream predictive utility. Robustness tests with randomly swapped and outof-distribution rationales indicate partial robustness, suggesting that semantic alignment contributes to performance gains. Overall, our results provide insights for selecting and reporting explanations that influence model behavior in practical prompting pipelines. ∗ Corresponding author 1 Introduction Natural language explanations (NLEs), also referred to as free-text rationales, have emerged as a prominent approach to improving the interpretability of language model (LM) predictions (Cambria et al., 2023; Wiegreffe & Marasovic, 2021; Bhan et al., 2024; Dhaini et al., 2025) . As natural-language justifications for model predictions, NLEs are widely used to communicate model reasoning 1 to users. Beyond interpretability, however, NLEs may also serve a functional role: recent work suggests that providing explanations during inference can improve model performance (Dhaini et al., 2025) . Human-annotated rationales are often considered a gold standard, but they are expensive, slow to obtain, and subject to annotation bias and inconsistency (Yao et al., 2023; Hartmann & Sonntag, 2022) . An alternative is to generate explanations automatically, either via self-explanations, in which the model justifies its own predictions, or by prompting an auxiliary LLM to generate rationales (Mishra et al., 2024; Wang et al., 2025; Wei Jie et al., 2024) . In parallel, in-context learning (ICL) has emerged as one of the key LLM capabilities, enabling task adaptation via example-driven prompts without parameter updates (Liu et al., 2021) .

FAQs

How does the study compare different natural language explanation types in in-context learning?

It evaluates four instruction-tuned models across six benchmarks while varying explanation source (human, self-generated, external LLM) and explanation selection (random vs faithfulness-based filtering).

What overall trends are observed when explanations are added to few-shot prompts?

On classification-style benchmarks, adding NLEs to few-shot prompts often improves accuracy compared with few-shot prompting without explanations.

How do faithfulness-based selection results differ across tasks and metrics?

Average gains from faithfulness-based selection of self-generated explanations are small, but performance can increase or decrease depending on the evaluation metric, task, and model. Different faithfulness metrics can select different examples and lead to substantially different downstream utility.

Sep 15, 2026

Uploaded by Aurora