Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix

Jul 16, 2026

Compact world models conditioned on a language goal can appear to ground spatial relations using explicit reference anchors, such as placing one object left of another. This work tests when such references truly support perception and control, and identifies a failure mode: instruction leakage. A goal-conditioned predictor achieves high relation readout accuracy largely by transcribing the instruction rather than sensing the scene. Withheld or counterfactual goals collapse accuracy, and directionally incorrect instructions follow the predictor. A goal-free dynamics fix and supervised anchor readout restore instruction-independent grounding across tabletop and BabyAI benchmarks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Abstract

Introduction

Problem Setup and Motivation

Measurement of the Grounding Intuition

Instruction Leakage Characterization

Goal-Free Dynamics Fix and Contributions

arXiv :2607 .06925v 1 [ cs .AI] 8 Jul 2026 Grounding Spatial Relations in a Compact World Model: Instruction Leakage and a Goal-Free Dynamics Fix Yufeng Wang∗1 Lu Wei∗1 Haibin Ling†2 1 Stony Brook University 2Westlake University ∗ Equal contribution †Corresponding author Abstract Compact world models that condition on a language goal promise to ground relations such as“put the red block left of the blue block” using a sparse set of explicit reference anchors. We ask when such references actually ground a relation, and identify a trap: a goal-conditioned predictor reaches a striking 0.90 relation-readout accuracy, yet this is instruction transcription, not perception. Withholding the goal collapses it to chance (0 .90 → 0 .27, three seeds) and a counterfactual instruction makes the predicted anchors follow the false instruction 94.5% of the time (true scene 2.3%; N =256) . Tested across three settings and a withintask ablation, our central claim characterizes the confound: instruction leakage occurs when the scored quantity is transcribable from the instruction (when the instruction names the answer) and is essentially independent of how predictive the non-instruction inputs are. Our tabletop and the external BabyAI benchmark leak, whereas a Language-Table forward-dynamics world model whose instruction names referents does not, until the instruction is augmented to name the direction; and degrading the action never increases leakage, the opposite of what predictor-competition predicts. The diagnosis prescribes the fix: keep the goal out of the dynamics (it belongs to the planner’s cost) and supervise the read path, recovering genuine, instruction-independent grounding (0 .88, identical with and without the goal) . The detection protocol and remedy apply to any goal-conditioned world model whose instruction names the scored quantity. 1 Introduction A central promise of world models for embodied control is that they let an agent imagine the consequences of its actions and plan against a goal (Ha & Schmidhuber, 2018 ; Hafner et al. , 2018 ; 2019 ; Hansen et al. , 2024 ; Assran et al. , 2025) . For relational goals such as “put X to the left of Y”, the agent must represent not just appearance but the inter-object spatial relation. A popular and compact recipe pairs a joint-embedding predictive (JEPA) latent with a small set of explicit, metric reference anchors and a language goal token (Maes et al. , 2026 ; Nam et al. , 2026): the anchors localize the referents, the latent carries the rest, and the goal conditions behavior. The intuition is that explicit coordinates give a relation a place to live that an entangled global latent does not. We set out to measure this intuition rather than assert it: when, and why, do explicit references help a compact world model represent and act on a spatial relation, and when does a text-conditioned latent already suffice? We build a controlled 2D relational tabletop, a diagnostic task in the spirit of CLEVR (Johnson et al. , 2016), in which referential ambiguity is a tunable knob (the number of duplicate distractors), train a ladder of models that vary the representation and the supervision, and evaluate them both as representations (can a readout recover the relation?) and as controllers (can a planner achieve the relation?) . The study turned on a confound that is worth stating up front because it is easy to fall into. A goalconditioned model appears to ground relations extremely well: a geometric readout from its predicted anchors reaches 0.90 accuracy and is, deceptively, robust to ambiguity. That number is almost entirely an artifact: because the instruction names the relation being scored, the predictor transcribes the instruction rather than perceiving the scene. Two controls, withholding the goal and substituting a counterfactual one, make this precise; the apparent grounding evaporates under both. Generalizing the phenomenon into a falsifiable characterization is

FAQs

How does the proposed “goal-free dynamics” fix mitigate the leakage?

It removes the goal from the dynamics predictor and instead injects the goal via the planner’s cost, while supervising the anchor read path so anchors encode the relation independently of the instruction.

Jul 16, 2026

Upload by Theodore