When Is Routing Meaningful Diversity and Robustness in Language Model Societies

Jul 16, 2026

Upload by Quinn

Routing policies for multi-model language systems are often judged only by downstream task accuracy and inference cost. This work shows routing is meaningful only when actor societies are behaviorally differentiated and router choices are stable to superficial surface-form variation. High accuracy can still fail meaningfulness if routers operate over redundant societies or route inconsistently. The paper adapts Hierarchic Social Entropy (HSE) for diversity and adds a perturbation-based robustness metric. Experiments on EmbedLLM and RouterBench reveal diminishing returns with small curated societies, diverging accuracy from meaningful robustness.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

Introduction

Related Work

LLM Routing and Model Selection

When is Routing Meaningful? Diversity and Robustness in Language Model Societies Fantine Huot Michael Kaisers Mirella Lapata Google DeepMind {fantinehuot,mkaisers, [lapata}@google.com](lapata}@google.com) arXiv :2607 .09197v1 [ cs .MA] 10 Jul 2026 Abstract Routing policies for multi-model systems are evaluated almost exclusively on task accuracy and inference cost. We argue that two properties, orthogonal to performance, determine whether routing is meaningful. First, the society of actors must be behaviourally differentiated: if all actors respond identically, routing is vacuous. Second, the routing policy must bestable: surface-form variants of a query should be assigned to the same actor. High task accuracy is compatible with violating both properties, since a router can operate over a redundant society or assign queries inconsistently, preventing specialisation regardless of performance. We adapt Hierarchic Social Entropy (HSE) to language-model societies and introduce a perturbation-based robustness metric to diagnose these failure modes. Applied to EmbedLLM and RouterBench, we find that HSE exhibits strong diminishing returns, suggesting that a curated subset of fewer than ten agents recovers most available diversity in a large pool—a practical coreset heuristic for society design. We further find that KNN routers gain accuracy from specialist societies but collapse in robustness under perturbation, while prompted routing remains stable across all perturbation types—illustrating that accuracy and meaningfulness can sharply diverge. 1 Introduction Routing is becoming a central mechanism for coordinating systems composed of multiple language models, tools, or agents (Wu et al., 2024 ; Hong et al., 2024 ; Li et al., 2023 ; Yue et al., 2025) . Existing work on LLM routing largely evaluates routers by downstream utility: whether they improve task accuracy, reduce cost, or select the best-performing model for a query (Chen et al., 2024 ; Ong et al., 2025 ; Hu et al., 2024 ; Jiang et al., 2023) . A router is not merely a predictor of task performance; it is also the coordination mechanism that determines how work is distributed across a society of actors. Its usefulness, therefore, depends on two structural properties. First, the society must contain behaviourally differentiated actors: if all actors respond in the same way, routing is vacuous (Balch, 2000 ; Bettini et al., 2025) . Second, the routing policy should be robust to superficial variation: semantically-equivalent variants should not be sent to different actors merely because of changes in spelling, syntax, or wording (Sclar et al., 2024) . Without such stability, actors cannot reliably specialise. To characterise the space of choices available to a router, we adapt Hierarchic Social Entropy (Balch, 2000) to language-model societies, measuring behavioural diversity from model outputs rather than parameters (Stanley et al., 2019) . To characterise the stability of the router’s choices, we introduce a perturbation-based robustness metric that measures whether semantically related inputs are assigned to the same actor. We validate HSE on two benchmarks, EmbedLLM (Zhuang et al., 2025) and RouterBench (Hu et al., 2024), comparing a default society of real-world language models against synthetic societies of purpose-designed experts varying in role and expertise overlap. We find that specialist societies achieve substantially higher HSE than pools of real-world models of equivalent size, suggesting that behavioural diversity in deployed model pools is lower than commonly assumed. We further observe strong diminishing returns: fewer than ten agents suffice to capture most available diversity in EmbedLLM, and four agents in RouterBench, providing a practical heuristic for society design. Interestingly, higher HSE does not imply higher robustness: KNN routers achieve their best accuracy on specialist societies but their worst robustness, while prompted

FAQs

Why can a routing policy be “meaningless” even when task accuracy is high?

Routing can remain meaningful only if actors are behaviorally differentiated and the router assigns semantically equivalent queries consistently. Accuracy alone can be high when the society is redundant or the router routes inconsistently, preventing real specialization.

What two structural properties determine when routing is meaningful?

First, the society of actors must be behaviorally differentiated; identical behavior makes routing vacuous. Second, routing policies must be stable, so surface-form variants of a query are routed to the same actor.

How do HSE and the proposed robustness metric help diagnose routing failure modes?

The paper adapts Hierarchic Social Entropy (HSE) to measure behavioral diversity from model outputs, and introduces a perturbation-based robustness metric to test whether semantically related inputs are assigned to the same actor. Together, they reveal cases where diversity and stability diverge.

Jul 16, 2026

Upload by Quinn