CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation

Aug 23, 2026

Upload by Eden

Lexical semantics studies word senses across contexts and the semantic relations between meanings of different words. Contextualized Language Models provide context-sensitive representations for lexical meaning, but prior work focused on Word-in-Context compares occurrences of the same lemma, limiting inter-word information. This paper proposes Concept Differentiation and introduces a SemCor-derived dataset, then fine-tunes multiple models to produce Concept-Aligned Embeddings (CALE). Experiments across lexical semantic tasks show strong multi-purpose performance and embedding space reorganization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Abstract

Introduction

From Word-in-Context to Concept Differentiation

Concepts vs. senses and desired embedding properties

Task and dataset SPCD

CALE model and evaluations

CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation Bastien Liétard1 and Gabriel Loiseau1,2 , 1University of Lille, Inria, CNRS, Centrale Lille, UMR 9189-CRIStAL, F-59000 Lille, France 2Hornetsecurity, Hem, France [first_name.last_name@inria.fr](first_name.last_name@inria.fr) Authors contributed equally. Abstract Lexical semantics is concerned with both the multiple senses a word can adopt in different contexts, and the semantic relations that exist between meanings of different words. To investigate them, Contextualized Language Models are a valuable tool that provides contextsensitive representations that can be used to investigate lexical meaning. Recent works like XL-LEXEME have leveraged the task of Wordin-Context to fine-tune them to get more semantically accurate representations, but Word-inContext only compares occurrences ofthe same lemma, limiting the range of captured information. In this paper, we propose an extension, Concept Differentiation, to include inter-words scenarios. We provide a dataset for this task, derived from SemCor data. Then we fine-tune several representation models on this dataset. We call these models Concept-Aligned Embeddings (CALE) . By challenging our models and other models on various lexical semantic tasks, we demonstrate that the proposed models provide efficient multi-purpose representations of lexical meaning that reach best performances in our experiments. We also show that CALE’s fine-tuning brings valuable changes to the spatial organization of embeddings. 1 Introduction Research in computational lexical semantics has relied on contextualized embeddings to study word meaning in context (Neidlein et al., 2020 ; Chronisand Erk, 2020 ; Apidianaki and Garí Soler, 2021 ; Yu and Xu, 2023 ; Li et al., 2024), with applications extending beyond traditional tasks, such as in debate modeling (Garí Soler et al., 2023) and political discourse analysis (Boholm et al., 2024) . However, most approaches use pre-trained models like BERT (Devlin et al., 2019) or XLM-R (Conneau et al., 2020) without fine-tuning specifically for lexical semantics, due to limited annotated data. While several models are fine-tuned for the Word-in-Context (WiC) task (Pilehvar and Camacho-Collados, 2019 ; Liu et al., 2021 ; Cassotti et al., 2023 ; Mosolova et al., 2024), WiC in its traditional definition only captures within-word meaning variation (i.e. the word’s senses) and ignores inter-word semantic relations. Consequently, models trained on WiC may struggle to capture the broader structure of lexical meaning. To reach a higher perspective, the focus must shift from word senses to semantic concepts. Throughout this paper, we distinguish senses and concepts as follows: a word sense is a conventionalized way to use a particular lemma in relevant contexts, it characterizes the use of a specific word for an intended meaning; concepts are mental representations of categories of objects, events, acts and ideas, and we use words to refer to them (Murphy, 2002) . In context, the meaning of a word is the concept it refers to, and a word sense is a pointer to a concept. Our concepts are equivalent to Wikipedia entries, or WordNet synsets (Miller, 1995) as made in Liétard et al. (2024) . In this paper, we are interested in the use of contextualized models to obtain multi-purpose (as opposed to “task-specific”) vector representation of word meaning. Word occurrences that instantiate the same concept in respective contexts should have similar representations, while occurrences of words that refer to different concepts should have dissimilar embeddings. A model with this desired property would be useful in studies of the mapping between word forms and concepts, like Haspelmath (2023) for instance, or in any computational lexical semantic applications mentioned above. We focus on synonymy (different words having the same meaning, instantiating the same concept) a

FAQs

What limitation of Word-in-Context motivates Concept Differentiation?

Word-in-Context traditionally compares occurrences of the same lemma, so it captures within-word sense variation but largely ignores inter-word semantic relations. Concept Differentiation extends the comparison to both same-lemma and cross-lemma pairs.

How is Concept Differentiation defined in the paper?

Given two word occurrences, the task determines whether they instantiate the same semantic concept. It is formulated as a binary classification over Concept or Frame Induction.

What are CALE (Concept-Aligned Embeddings) and what do the experiments show?

CALE is a family of token-level models fine-tuned for Concept Differentiation using the SPCD dataset. The results show strong performance across multiple lexical semantic tasks and indicate that fine-tuning shifts embedding organization from lemma-centric to concept-centric representations.

Aug 23, 2026

Upload by Eden