Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation

Jul 16, 2026

Upload by Aurelia

Language grounding in embodied world models raises a core question: how can natural language interface with a discrete symbol bottleneck without corrupting the symbol system? Experiments show that allowing language gradients into a discrete layer forces a structural trade-off—vanilla Gumbel-softmax collapses symbols, while anti-collapse variants preserve diversity but fail to learn semantic labels (≤9.4% accuracy). No GumbelBottleneck variant satisfies both goals. A minimal three-layer constraint set blocks gradients, adds a gradient-free memory-table semantic channel, and resolves symbol collisions via DPMeans, achieving 97.2% grounding accuracy and 0% collapse across many runs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

Introduction

Gradient–Discrete Interface Trade-off

Minimal Architectural Fix

Conclusion

arXiv :2607 .083 12v 1 [ cs .LG] 9 Jul 2026 Write-Protected Discrete Bottlenecks for Language-Grounded World Models Jiayi Fang Shanghai University of Finance and Economics Abstract How should language interface with a world model’s discrete symbol system? The dominant paradigm—end-to-end injection of LLM/VLM features into robot world models (RT-2, Octo, PaLM-E)—implicitly assumes that language gradients can directly shape physical symbol representations. We ask whether this assumption is safe, find that it is not, and characterize the minimal architectural constraint that prevents the failure. Any language gradient entering a discrete symbol bottleneck forces a structural trade-off: the vanilla Gumbel-softmax estimator collapses to 2/64 symbols, while five anti-collapse strategies maintain diversity but fail to learn semantic labels (all ≤ 9.4% accuracy) . No GumbelBottleneck variant achieves both objectives simultaneously. This is a structural limitation, not an optimization problem: language gradients cannot enter a discrete symbol layer. The minimal fix has three layers: (1) cut the gradient chain (z .detach()), preventing language signals from reaching the symbol bottleneck; (2) provide a gradient-free semantic channel—a nonparametric Memory Table (Dict[symbol → Counter [label]], zero parameters, zero gradients) where co-occurrence counting replaces gradient-based binding; (3) handle symbol collisions via DPMeans streaming clustering for automatic sub-cluster splitting. Layer 1 alone prevents collapse but cannot bind semantics. Layer 2 enables binding but fails when multiple objects share a symbol. All three layers together form a minimal complete set: 97.2% grounding accuracy with all three vs. 22.2% without Layer 3 . Across two experiments spanning 74 independent runs, we demonstrate: (1) end-to-end alternatives face a structural ceiling—vanilla Gumbel-softmax collapses while anti-collapse strategies maintain diversity but fail to learn; (2) the three-layer fix generalizes across three encoder architectures (CNN, V-JEPA 300M, CLIP ViT-L), two environments (grid world, MuJoCo 3D desktop), and three texture conditions—zero collapse in all 32 seeds, with the blackboard achieving 79–100% semantic binding. We further discuss the environment loop (language → action → re-perception) as a qualitative mechanism by which language may indirectly shape physical experience. The fix trains fewer than 2M parameters, requires no LLM fine-tuning, and adds zero computational overhead for semantic binding. Our results challenge the end-to-end scaling paradigm: the bottleneck isnot LLM capacity, but whether the architecture separates physical perception from language processing. 1 Introduction If a robot navigates to a red cube after hearing “go to the red cube,” did language teach it what red means? Current approaches in embodied AI answer yes—large language and vision-language models (LLMs/VLMs) are injected end-to-end into robot world models, with the expectation that richer language representations will improve physical understanding [2, 4, 7] . This paradigm has produced impressive demonstrations, but it conflates two functions we show must be architecturally separated: physical symbol formation and language-driven semantic binding. Worse, it embeds a scaling assumption—that larger LLMs lead to better physical grounding—which our experiments contradict. We test the end-to-end assumption directly and find that it fails systematically. Any language gradient entering a discrete symbol bottleneck forces a structural trade-off: the vanilla Gumbel-softmax estimator collapses to 2.2/64 symbols (4/5 seeds), while four anti-collapse strategies (high temperature, low learning rate, spectral norm, entropy bonus) maintain moderate diversity (4–17/64 depending on strategy) but fail to learn semantic labels (accuracy ≤ 9.2%, barely above chance at 2 . 8%) . No GumbelBottleneck variant simultaneously achieves high diversity

FAQs

What problem does the paper study about language-grounded world models?

It examines how natural language should connect to a world model’s discrete symbol system, and whether end-to-end gradient injection is architecturally safe. The study finds that letting language gradients enter the discrete bottleneck causes structural failure modes.

Why do language gradients entering a discrete symbol bottleneck fail?

The interface creates an unavoidable structural trade-off: vanilla Gumbel-softmax collapses to very few symbols, while anti-collapse methods maintain diversity but cannot learn semantic labels, staying near chance accuracy. No tested GumbelBottleneck variant achieves both diversity and semantic learning together.

What is the minimal architectural fix proposed by the paper?

The fix uses three layers: a gradient cut (z.detach()) to prevent language signals reaching the symbol bottleneck, a gradient-free semantic channel via a nonparametric memory table for symbol-to-label co-occurrence counting, and DPMeans streaming clustering to split clusters when labels collide on a symbol. Together they prevent collapse and enable semantic binding.

Jul 16, 2026

Upload by Aurelia