ICLR-2026-routing-manifold-alignment-improves-generalization-of-mixture-of-experts-llms-Paper-Conference - research results

Sep 2, 2026

Upload by Rhys

A comparative evaluation table reports performance across multiple language and reasoning benchmarks for mixture-of-experts LLM variants. The results contrast baseline models with tuning strategies such as ICL, C3PO, Router Tuning, Oracle Tuning, Prefix Tuning, Prompt Tuning, and Dense BP. The method “RoMA (Ours)” shows consistent gains over baselines and several tuning baselines on datasets including MMLU, HellaSwag, ARC-C, ARC-E, PIQA, WinoGrande, BoolQ, and GSM8K, with an average score column summarizing improvements. Additional sections study effects under different active-parameter regimes and model sizes.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Experimental Results

Benchmark Scores Across Methods

Active-Parameter Regimes and Model Sizes

| Method | MMLU | Hella
Swag | ARC-C | ARC-E | PIQA | Wino
Grande | BoolQ | GSM8K | Avg | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | DeepSeekMoE-16B-A3B | | | | | | | | | | | Base model | 46.2 | 78.0 | 50.3 | 73.8 | 79.9 | 70.1 | 72.3 | 62.2 | 66.6 | | Oracle | 63.8 | 92.5 | 70.8 | 85.2 | 90.3 | 82.1 | 83.2 |
71.8 | 80.0 | | ICL | 49.0 | 81.6 | 56.3 | 76.2 | 81.4 | 72.3 | 75.8 | 65.7 | 69.8 | | C3PO | 55.4 | 85.7 | 61.6 | 80.7 | 85.8 | 77.5 | 78.2 | 68.5 | 74.2 | | | | | | | | | | | | | Router Tuning | 49.3 | 81.5 | 57.2 | 76.6 | 82.0 | 73.8 | 74.5 | 64.8 | 70.0 | | Oracle Tuning | 54.2 | 84.3 | 60.1 | 79.5 | 84.0 | 76.0 | 77.5 | 66.2 | 72.7 | | Prefix Tuning | 47.8 | 77.9 | 52.4 | 73.8 | 79.2 | 70.3 | 73.1 | 64.8 | 67.4 | | Prompt Tuning | 49.3 | 78.6 | 55.1 | 74.7 | 80.5 | 72.0 | 74.2 | 65.5 | 68.7 | | Dense BP | 50.1 | 80.2 | 54.8 | 77.3 | 81.7 | 74.2 | 76.1 | 63.9 | 69.8 | | RoMA (Ours) 56.8 87.9 61.4 81.5 85.2 76.8 80.6 67.4 74.7 | | | | | | | | | | | OLMoE-7B-A1B | | | | | | | | | | | Base model | 57.8 | 77.9 | 51.3 | 79.8 | 80.7 | 72.2 | 75.4 | 45.5 | 67.6 | | Oracle | 72.2 | 91.5 | 74.8 | 91.4 | 93.6 | 87.7 | 84.5 |
53.2 | 81.1 | | ICL | 60.3 | 80.6 | 58.1 | 82.5 | 83.6 | 76.8 | 78.9 | 48.5 | 71.2 | | C3PO | 65.5 | 85.3 | 66.3 | 87.4 | 88.0 | 82.7 | 79.6 | 50.8 | 75.7 | | | | | | | | | | | | | Router Tuning | 63.2 | 81.7 | 62.5 | 83.8 | 80.9 | 75.3 | 77.8 | 47.2 | 71.6 | | Oracle Tuning | 66.8 | 84.2 | 65.4 | 86.1 | 86.2 | 80.5 | 79.9 | 49.0 | 74.8 | | Prefix Tuning | 59.3 | 78.2 | 54.5 | 80.4 | 82.1 | 73.5 | 76.8 | 46.7 | 68.9 | | Prompt Tuning | 59.7 | 79.5 | 55.9 | 81.3 | 82.4 | 74.1 | 77.2 | 47.3 | 69.7 | | Dense BP | 61.8 | 82.4 | 57.3 | 84.1 | 83.9 | 76.9 | 75.2 | 48.1 | 71.2 | | RoMA (Ours) 69.0 86.7 67.2 88.0 85.8 81.8 81.7 49.4 76.2 | | | | | | | | | | | Qwen3-30B-A3B | | | | | | | | | | | Base model | 74.2 | 68.5 | 56.8 | 84.3 | 78.5 | 65.2 | 81.3 | 83.4 | 74.0 | | Oracle | 82.5 | 80.3 | 69.2 | 92.6 | 87.4 | 77.3 | 90.5 |
90.9 | 83.8 | | ICL | 75.8 | 70.7 | 59.3 | 86.1 | 80.2 | 67.8 | 83.5 | 84.7 | 76.0 | | C3PO | 77.9 | 74.1 | 63.4 | 88.1 | 81.7 | 71.9 | 85.4 | 86.0 | 78.6 | | | | | | | | | | | | | Router Tuning | 75.3 | 70.3 | 60.1 | 85.7 | 79.8 | 68.5 | 82.8 | 84.2 | 75.8 | | Oracle Tuning | 77.2 | 73.5 | 62.8 | 87.6 | 81.3 | 71.2 | 84.9 | 85.5 | 78.0 | | Prefix Tuning | 74.5 | 68.9 | 57.9 | 84.8 | 79.1 | 66.3 | 82.1 | 83.8 | 74.7 | | Prompt Tuning | 75.0 | 69.6 | 58.6 | 85.2 | 79.6 | 67.0 | 82.7 | 84.0 | 75.2 | | Dense BP | 76.1 | 71.4 | 59.8 | 86.5 | 80.5 | 69.2 | 83.8 | 84.9 | 76.5 | | RoMA (Ours) | 78.8 | 74.8 | 65.5 | 88.6 | 83.1 | 73.8 | 85.1 | 86.3 | 79.5 | | MMLU | | Hella
Swag | ARC-C ARC-E PIQA | | | Wino
Grande | BoolQ GSM8K | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | ∼1B active parameters | | | | | | | | | | Llama3.2-1B | 27.4 | 57.9 | 32.1 | 53.9 | 72.4 | 57.4 | 63.7 | 39.4 | | OLMo-1B | 24.1 | 61.8 | 29.6 | 55.7 | 75.6 | 56.8 | 64.2 | 28.5 | | OLMoE-7B-A1B | 57.8 | 77.9 | 51.3 | 79.8 | 80.7 | 72.2 | 75.4 | 45.5 | | ∼3B active parameters | | | | | | | | | | Gemma2-3B | 43.7 | 66.3 | 58.4 | 75.2 | 71.8 | 64.5 | 73.1 | 41.4 | | DeepSeekMoE-16B-A3B | 46.2 | 78.0 | 50.3 | 73.8 | 79.9 | 70.1 | 72.3 | 62.2 | | Qwen3-30B-A3B | 74.2 | 68.5 | 56.8 | 84.3 | 78.5 | 65.2 | 81.3 | 83.4 | | ∼7-8B active parameters | | | | | | | | | | Qwen2-7B | 53.4 | 74.9 | 45.8 | 69.7 | 77.2 | 68.1 | 84.8 | 79.9 | | Mistral-7B | 59.6 | 81.0 | 53.8 | 79.6 | 82.2 | 74.0 | 68.1 | 37.9 | | Llama3.1-8B | 57.7 | 77.9 | 48.7 | 80.8 | 81.4 | 73.5 | 81.9 | 49.6 | | ∼13-14B active parameters | | | | | | | | | | Llama2-13B | 53.8 | 78.6 | 50.1 | 74.5 | 79.1 | 70.1 | 75.7 | 35.2 | | Vicuna-13B | 51.3 | 76.2 | 47.4 | 72.8 | 78.0 | 68.2 | 71.5 | 32.2 | | Qwen1.5-14B | 66.7 | 81.5 | 58.0 | 85.3 | 82.1 | 76.9 | 81.3 | 58.4 | | ∼27-34B active parameters | |

FAQs

What evaluation benchmarks are reported in the table?

The table includes MMLU, HellaSwag, ARC-C, ARC-E, PIQA, WinoGrande, BoolQ, GSM8K, and an average column summarizing overall performance.

Which tuning strategies are compared against baseline models?

The document compares ICL, C3PO, Router Tuning, Oracle Tuning, Prefix Tuning, Prompt Tuning, and Dense BP, alongside base models and RoMA (Ours).

How does RoMA (Ours) perform relative to other methods?

RoMA (Ours) achieves higher scores than the base model and several tuning baselines across multiple benchmarks, and it shows improved average performance in the reported comparisons.

Sep 2, 2026

Upload by Rhys