Unlocking Multilingual Reasoning Capabilities of LLMs and LVLMs via Representation Engineering

Jul 16, 2026

Upload by Anda

Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) show strong reasoning overall, but their performance drops sharply in low-resource languages compared with English, creating fairness challenges in multilingual deployment. Existing solutions require costly multilingual training or translation-dependent prompting with sensitivity to quality and latency. MRRE introduces a training-free inference-time approach that injects two precomputed vectors to steer non-English reasoning into an English-aligned space and restore target-language consistency, improving benchmarks by up to 7.54%.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

Abstract

Introduction

Motivation and prior work

Proposed MRRE method

Related Works

Multilingual Foundation Models

Unlocking Multilingual Reasoning Capabilities of LLMs and LVLMs via Representation Engineering Qiming Li 1 ,∗ , Xiaocheng Feng 1 ,2 , Yixuan Ma 1 * , Ruihan Chen 1 , Zihe Tong 1 , Zekai Ye 1 Xiachong Feng3 , Libo Qin4 , Haoyu Ren5 , Kun Chen5 , Yunfei Lu5 , Dandan Tu5 , Bing Qin 1 ,2 1Harbin Institute of Technology 2Peng Cheng Laboratory 3The University of Hong Kong 4Harbin Institute of Technology, Shenzhen 5Huawei Technologies Co., Ltd [qmli@ir.hit.edu.cn](qmli@ir.hit.edu.cn) arXiv :2511 .23231v2 [ cs .CV] 11 Jul 2026 Abstract Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) demonstrate strong reasoning capabilities, yet their performance in English significantly outperforms that in low-resource languages, raising fairness concerns in multilingual applications. Existing approaches either rely on costly multilingual training or employ prompting with external translation tools, both of which are resource-intensive and sensitive to translation quality. To address these limitations, we propose a training-free inference-time method to enhance Multilingual Reasoning capabilities via Representation Engineering (MRRE) without using any additional training data or tools. MRRE sequentially injects two precomputed vectors at specific layers during inference processing: cross-lingual reasoning enhancement vectors, which steer non-English reasoning representations toward English space to unlock multilingual reasoning, and target-language output anchoring vectors, which restore the distribution of the target language to preserve input–output language consistency. Comprehensive experiments across six advanced LLMsand LVLMs on four reasoning benchmarks demonstrate MRRE consistently enhances nonEnglish reasoning by an average gain of 5.48% and up to 7.54% in low-resource languages (e.g., Thai and Swahili), while improving inputoutput language consistency by 3.78% . 1 Introduction With the rapid development of Large Language Models (LLMs) and Large Vision-Language Models (LVLMs), foundational models such as Llama-3 . 1-8B-Instruct (Grattafiori et al., 2024) and Qwen2 .5-VL-7B-Instruct (Team, 2025) have demonstrated impressive capabilities in complex reasoning. However, high-resource languages such as English exhibit substantially stronger reasoning *Equal contribution. Figure 1: MRRE adopts a two-stage intervention strategy to unlock multilingual reasoning capabilities. capabilities compared to low-resource languages, raising concerns about fairness in multilingual applications under global deployment. To address the above issues, prior works primarily focus on two directions to enhance non-English reasoning capabilities: (1) Data-driven training methods, which align multilingual embeddings (Arora et al., 2024) or construct multilingual reasoning datasets for instruction tuning (Fan et al., 2025), but inevitably depend on expensive data and incur considerable computational costs. (2) Prompting-based methods, which rely on external translation tools or models (Khandelwal et al., 2024 ; Liu et al., 2024), but are highly sensitive to translation quality and prompt design, accompanied by high latency. Moreover, existing methods are rarely effective across both LLMs and LVLMs. Therefore, establishing a unified and efficient paradigm for enhancing multilingual reasoning across both LLMs and LVLMs remains further exploration. Prior studies (Zhao et al., 2024 ; Tang et al., 2025 ; Qin et al., 2025 ; Li et al., 2025a,b ; Zhao et al., 2025) revealed the internal mechanism of multilingual reasoning: hidden states are transformed into high-resource language representations (e.g., English) in early layers, then exploited for reasoning from middle to later layers, and finally restored target language features in late layers. However, as shown in Figure 2, we observe that hidden (1) Hidden state visualization of Qwen2 .5-7B-Instruct (2) Hidden state visualization of Qwen2 .5-VL-7B-Instruct Figure 2: t-SN

FAQs

What problem does MRRE address for multilingual reasoning in LLMs and LVLMs?

MRRE targets the gap where reasoning is much stronger in English than in low-resource languages, which raises fairness concerns for global multilingual applications.

How does MRRE improve multilingual reasoning without additional training data or tools?

MRRE performs training-free inference-time intervention by sequentially injecting two precomputed vectors at specific layers: enhancement vectors to steer non-English representations toward English reasoning space, and anchoring vectors to restore target-language distribution for input-output consistency.

What performance improvements does MRRE achieve on multilingual reasoning benchmarks?

Experiments across six LLMs and LVLMs on four benchmarks show an average gain of 5.48% for non-English reasoning and up to 7.54% in low-resource languages, along with a 3.78% improvement in input-output language consistency.

Jul 16, 2026

Upload by Anda