RARE - Router-Agnostic Representation Engineering - Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models

Sep 15, 2026

Uploaded by Ophelia

Representation engineering provides a lightweight way to steer language-model behavior by editing intermediate hidden states, but directly applying it to Mixture-of-Experts (MoE) causes a structural mismatch that hurts steering reliability. Systematic empirical studies confirm the failure mode and show that preserving clean routing can substantially recover performance, with routing depending more on semantic content than on behavior-only changes. Building on these insights, RARE is proposed as a router-agnostic framework that removes router-visible components via null-space projection and corrects routing drift in downstream layers.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Abstract

Introduction

Background on MoE architectures

Representation engineering and steering challenge

Research questions (RQ1-RQ3)

Routing drift and controlled probes

Proposed framework: RARE

Perturbation estimators and evaluation scenarios

RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models Zhibo Zhang Huazhong University of Science and Technology Wuhan, China [zhibozhang0312@gmail.com](zhibozhang0312@gmail.com) Zhen Ouyang Huazhong University of Science and Technology Wuhan, China [cookingmaster0920@gmail.com](cookingmaster0920@gmail.com) arXiv : 2608 .2 1236v 1 [cs .CL] 2 1 Aug 2026 Ling Shi AIDX TECH PTE. LTD. Singapore [ling.shi@aidxtech.com](ling.shi@aidxtech.com) Kailong Wang∗ Huazhong University of Science and Technology, Wuhan, China National University of Singapore, Singapore [wangkl@hust.edu.cn](wangkl@hust.edu.cn) Abstract Representation engineering offers a lightweight means of controlling language-model behavior by modifying intermediate hidden states, but its direct application to Mixture-of-Experts (MoE) models introduces a structural mismatch. We first verify this failure mode through a series of empirical studies and find that preserving clean routing substantially recovers steering performance and that routing is more sensitive to semantic content than to behavioral changes under controlled content. Motivated by these findings, we introduce RARE, a router-agnostic representation engineering framework for MoE language models. RARE projects arbitrary behavioral perturbations onto the null space of the router matrix, thereby removing router-visible components, and further corrects routing drift propagated to selected downstream layers. To decide the best perturbation estimator in this framework, we evaluate five estimators on six heterogeneous open-weight MoE models across three steering scenarios: harmfulness, truthfulness, and factual editing. On harmfulness steering, RARE reaches an average attack success rate of 53.3% while retaining 67.8% MMLU accuracy, yielding a stronger aggregate effectiveness–utility tradeoff than baselines. It further improves average TruthfulQA MC1 accuracy from 41.0% to 58.6% and CounterFact efficacy from 16.8% to 96.3% . These results support routing consistency as an important architectural consideration for adapting representation engineering to MoE models. 1 Introduction Large Language Models (LLMs) are increasingly built with Mixture-of-Experts (MoE) architectures, where each token is routed to a sparse subset of feed-forward experts. This conditional-computation paradigm enables models to scale capacity while keeping inference cost tractable, and has become a central design choice in recent large-scale systems such as GPT (OpenAI, 2025), DeepSeek (DeepSeek-AI, 2024), and Qwen (Qwen Team, 2025) . Representation engineering provides a lightweight mechanism for controlling languagemodel behavior by intervening on intermediate hidden states, and has achieved notable performance on dense LLMs (Zou et al., 2023a; Turner et al., 2024; Arditi et al., 2024; Stoehret al., 2024) . However, conditional computation architecture makes steering MoE models especially challenging. Directly transferring paradigms designed for dense models to MoEs introduces an architectural complication: the steering direction estimated during clean inference would disrupt the router logits, redirecting tokens to different expert sets ∗ Corresponding Author. compared with the original clean states, thereby leading to poor steering effects. Figure 1a illustrates the steering failure mode caused by this mismatch. Recent MoE-specific steering studies treat expert allocation as the intervention interface by immediately controlling router logits to activate only intended experts (Fayyaz et al., 2025; Lai et al., 2025) . While such interventions can effectively alter model behavior, they modify the native input-conditioned computation path and may consequently degrade response relevance and general capability. The central challenge is therefore how to coordinate representation-level control with the conditional routing mechanism of MoE models. We investigate this chal

FAQs

Why does representation engineering fail when applied directly to Mixture-of-Experts models?

MoE routing is input-conditioned, so a steering direction estimated on clean states can alter router logits and redirect tokens to different expert sets, producing poor steering effects.

What does the paper find about how routing changes relate to semantic versus behavioral changes?

Across controlled probes, routing changes far less when response behavior changes under a fixed input query topic, indicating routing mainly reflects input semantics rather than behavior differences.

How does RARE achieve router-agnostic steering in MoE language models?

RARE projects arbitrary behavioral perturbations onto the null space of the router matrix to remove router-visible components, and further corrects routing drift propagated to selected downstream layers.

Sep 15, 2026

Uploaded by Ophelia