Examining Multilingual Embedding Models Cross-Lingually Through LLM-Generated Adversarial Examples

Oct 3, 2026

The evaluation of cross-lingual semantic search models is often constrained by existing datasets focused on information retrieval and semantic textual similarity. CrossLingual Semantic Discrimination (CLSD) introduces a lightweight task that uses parallel sentences plus a Large Language Model (LLM) to generate adversarial distractors. CLSD tests whether an embedding model ranks the true parallel sentence above lexically similar but semantically misleading alternatives, with experiments for German–French news data.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

Introduction

Contributions

Related Work

Distractor Generation

Examining Multilingual Embedding Models Cross-Lingually Through LLM-Generated Adversarial Examples Andrianos Michail Simon Clematide Rico Sennrich University of Zurich {andrianos.michail,simon.clematide,sennrich}@cl.[uzh.ch](uzh.ch) arXiv :2502 .08638v4 [ cs .CL] 9 Oct 2025 Abstract The evaluation of cross-lingual semantic search models is often limited to existing datasets from tasks such as information retrieval and semantic textual similarity. We introduce CrossLingual Semantic Discrimination (CLSD), a lightweight evaluation task that requires only parallel sentences and a Large Language Model (LLM) to generate adversarial distractors. CLSD measures an embedding model’s ability to rank the true parallel sentence above semantically misleading but lexically similar alternatives. As a case study, we construct CLSD datasets for German–French in the news domain. Our experiments show that models finetuned for retrieval tasks benefit from pivoting through English, whereas bitext mining models perform best in direct cross-lingual settings. A fine-grained similarity analysis further reveals that embedding models differ in their sensitivity to linguistic perturbations.1 1 Introduction When selecting a model for semantic search in a specific domain and language pair, standard embedding benchmarks such as (Multilingual) MTEB (Muennighoff et al., 2023 ; Enevoldsen et al., 2025) provide only partial coverage. We argue that evaluation should be carried out directly on the target language pair and domain of interest. However, the lack of suitable datasets makes it difficult to reach well-founded decisions about applications, such as choosing an embedding model or deciding whether to use English as a pivot language. We propose a simple and efficient method for dataset generation and evaluation that simulates real-world challenges of cross-lingual semantic retrieval in large text collections. Our approach uses LLMs to create adversarial examples from parallel 1We release our code and datasets under AGPL- 3.0: [https://github.com/impresso/cross_lingual_](https://github.com/impresso/cross_lingual_)semantic_discrimination sentences, designed to test the cross-lingual embedding capabilities of multilingual models. Concretely speaking, four distractor sentences are generated for each parallel target sentence. We require these sentences to be very similar to the original target sentence in terms of syntactic structure and surface word forms, but to be semantically dissimilar. Our proposed task, Cross-Lingual Semantic Discrimination (CLSD), involves identifying the correct target sentence from among four distractor sentences in the target language, given an original sentence in the source language. To better understand how these distractor modifications influence semantic similarity both crosslingually and monolingually, we perform a finegrained analysis and correlate the semantic changes with linguistic properties at the level of part-ofspeech tags. Contributions • We propose a new task that emulates crosslingual semantic search within large text collections, requiring only a set of parallel sentences for dataset creation. • We publish four such datasets in the news domain for the German–French language pair. • We show that some embedding models perform better in direct cross-lingual retrieval, while others benefit from using English as a pivot language. • We provide a linguistically informed analysis that offers insights into cross-and monolingual semantic representations of multilingual models. 2 Related Work Cross-Lingual Semantic Search Cross-lingual semantic search is an umbrella term for tasks that | Original Sentence S | Distractor Sentence 1 D1 | Distractor Sentence 2 D2 | | --- | --- | --- | | Die Linkspartei beschließt in Bonn ihr Programm zur Europawahl.
(The Left Party adopts its program for the European elections in Bonn.) | Die Linkspartei beschließt in Bonn ihr Programm zur Bundestagswahl.

FAQs

What is Cross-Lingual Semantic Discrimination (CLSD)?

CLSD is a lightweight evaluation task that uses parallel sentences and an LLM to generate adversarial distractor sentences. It measures whether an embedding model ranks the true parallel sentence above semantically misleading alternatives.

How are adversarial distractors created in CLSD?

For each parallel target sentence, CLSD generates four distractor sentences. The distractors are designed to be very similar to the original in syntactic structure and surface word forms, while remaining semantically dissimilar.

What do the experiments suggest about English pivoting and model types?

Models fine-tuned for retrieval benefit from pivoting through English, while bitext mining models perform best in direct cross-lingual settings. The fine-grained analysis also shows models differ in sensitivity to linguistic perturbations.

Oct 3, 2026

Upload by Well Done