A Cognitively Motivated Multidimensional Framework for Evaluating Metaphor Explanations - Abstract and Evaluation Dimensions

Sep 15, 2026

Uploaded by Finn

Current evaluation of metaphor explanations often uses holistic quality ratings, providing limited insight into how explanation quality is structured and where human judgments converge or diverge. A cognitively motivated framework is introduced that decomposes metaphor explanation quality into six theoretically grounded dimensions. Results from a dense annotation study (11,200 ratings) show multidimensional quality, systematic annotator disagreement, and a shared cluster with two independent judgment axes. A feasibility study further indicates automatic evaluation can recover parts of this structure.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Abstract

1 Introduction

Metaphors and explanation challenges

Existing automatic metrics and limitations

Human-centered and multidimensional evaluation

Proposed evaluation dimensions

Criterion table and theoretical grounding

A Cognitively Motivated Multidimensional Framework for Evaluating Metaphor Explanations Ana Naveriani Constructor University Mehul Bhatt Örebro University Jakob Suchan Constructor University Antonio Lieto University of Salerno Stefano Zoia University of Turin Gian Luca Pozzato University of Turin CoDesign Lab EU – Artificial and Human Intelligence [codesign-lab.org](codesign-lab.org) arXiv :2608 . 15828v1 [ cs .CL] 16 Aug 2026 Abstract Current evaluation of metaphor explanations relies mainly on holistic quality ratings, revealing little about how explanation quality is structured or where human judgments agree and diverge. We introduce a cognitively motivated framework that decomposes metaphor explanation quality into six theoretically grounded dimensions. In a dense annotation study (11,200 ratings), we find that: (i) explanation quality is genuinely multidimensional; (ii) annotator disagreement is systematic rather than random; and (iii) the six dimensions collapse into a shared cluster and two independent axes of judgment. An exploratory feasibility study further shows that a standard automatic evaluation pipeline can recover parts of this structure, predicting the most discriminative dimensions well while its errors correlate human (dis)agreement. Together, these results suggest that multidimensional evaluation offers richer diagnostic insight than holistic ratings, and that automatic evaluators for open-ended generation tasks should be judged on how well they preserve the structure of human judgment. 1 Introduction Metaphors are a pervasive feature of human lan guage and thought (Lakoff and Johnson, 1980, 1999), enabling people to understand abstract concepts through more familiar and concrete experiences. For instance, consider the following passage: “Life is a journey where time is money and the mind works like a computer. But an Achilles’ heel may prove difficult to overcome, and a heart of stone can keep us from what truly matters.” While most speakers intuitively understand these expressions, explaining why they convey meaning is considerably more difficult (Glucksberg, 2001) . Effective explanations must reveal underlying conceptual mappings, account for emergent meaning, connect figurative language to familiar experiences, and provide sufficient context for interpretation. Consequently, metaphor explanation is not merely a linguistic task, but also a cognitive and communicative one. Research in cognitive linguistics has studied metaphors as a cognitive mechanism through which abstract concepts are understood via mappings from more concrete domains (Lakoff and Johnson, 1980, 1999) . Computational research has focused primarily on metaphor identification, interpretation, and generation (Shutova et al., 2010 ; Tsvetkov et al., 2014), while recent large language models have substantially improved the quality of ex-post generated metaphor explanations (Ichien et al., 2024) . However, comparatively little work has focused on their systematic evaluation. As such, evaluation remains a central challenge in Natural Language Generation (NLG) . Traditional automatic metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2020) often correlate poorly with human judgments for open-ended generation tasks (Schmidtova et al., 2024) . Consequently, recent work has increasingly emphasized human-centered and multidimensional evaluation frameworks (Howcroft et al., 2020 ; Celikyilmaz et al., 2021 ; van der Lee et al., 2021), while LLM-based evaluators have emerged as promising alternatives for automatic assessment (Zheng et al., 2023 ; Liu et al., 2023 ; Huang et al., 2025) . Furthermore, human disagreement is increasingly recognized as informative factor for evaluation (Uma et al., 2021) . Existing work on metaphor generation and explanation typically relies on holistic quality ratings or a small number of task-specific criteria (Chakrabartyet al.

FAQs

What problem do the authors address in evaluating metaphor explanations?

Holistic quality ratings reveal little about how explanation quality is structured or where human judgments agree and disagree. The work targets this gap with a dimension-based evaluation approach.

How is metaphor explanation quality modeled in the proposed framework?

It is decomposed into six theoretically grounded dimensions, capturing different aspects such as mapping, emergence of non-literal meaning, accessibility, and communicative purpose.

What do the annotation results from 11,200 ratings show?

Explanation quality is genuinely multidimensional, annotator disagreement is systematic rather than random, and the six dimensions reduce to a shared cluster plus two independent judgment axes.

Sep 15, 2026

Uploaded by Finn

A Cognitively Motivated Multidimensional Framework for Evaluating Metaphor Explanations - Abstract and Evaluation Dimensions_Page 1

Open in WPS to view all 11 pages