RoboStream Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics

Jul 16, 2026

Upload by Finn

Enabling reliable long-horizon robotic manipulation for open-world embodied intelligence requires persistent spatial and causal awareness. Vision-language model planners often treat each step as an isolated observation-to-action mapping, repeatedly reinferring geometry from raw pixels and lacking memory of how prior actions reshape the environment. As a result, perceptual errors accumulate, temporarily occluded objects are forgotten, and compounding failures cascade into precondition violations. RoboStream introduces STF-Tokens for geometric anchoring and a Causal Spatio-Temporal Graph for action-triggered state transitions, enabling object permanence under occlusion without extra training.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages41

Reading Time1 h 43 min

Introduction

Problem: Limitations of step-wise VLM planning

Proposed approach: RoboStream with STF-Tokens and CSTG

arXiv :2603 . 12939v2 [ cs .RO] 10 Jul 2026 RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics Yuzhi Huang 1 ,2∗‡♠ Jie Wu 1 ,2∗‡ Weijue Bu3∗ Ziyi Xiong2 ,4‡ Gaoyang Jiang5 Ye Li 1 Kangye Ji 1 ,2‡ Shuzhao Xie 1 Yue Huang6 Chenglei Wu2 Jingyan Jiang4† Zhi Wang 1† 1 Shenzhen International Graduate School, Tsinghua University 2 YuanxingGuangnian Robotics 3 China University of Mining and Technology 4 Shenzhen Technology University 5 Huazhong University of Science and Technology 6 Xiamen University Website: [https://robostream123.github.io/](https://robostream123.github.io/) Abstract. Enabling reliable long-horizon robotic manipulation is a crucial step toward open-world embodied intelligence. However, VLM-based planners treat each step as an isolated observation-to-action mapping, forcing them to reinfer scene geometry from raw pixels at every decision step while remaining unaware of how prior actions have reshaped the environment. Despite strong short-horizon performance, these systems lack the spatio-temporal reasoning required for persistent geometric anchoring and memory of action-triggered state transitions. Without persistent state tracking, perceptual errors accumulate across the execution horizon, temporarily occluded objects are catastrophically forgotten, and compounding failures lead to precondition violations that cascade through subsequent steps. In contrast, humans maintain a persistent mental model that continuously tracks spatial relations and action consequences across interactions rather than reconstructing them at each instant. Inspired by this human capacity for causal spatio-temporal reasoning with persistent memory, we propose RoboStream, a training-free framework that achieves geometric anchoring through Spatio-Temporal Fusion Tokens (STF-Tokens), which bind visual evidence to 3D geometric attributes for persistent object grounding, and maintains causal continuity via a Causal Spatio-Temporal Graph (CSTG) that records action-triggered state transitions across steps. This design enables the planner to trace causal chains and preserve object permanence under occlusion without additional training or fine-tuning. RoboStream achieves a 90 .5% success rate on long-horizon RLBench tasks and a 44 .4% success rate on challenging real-world block-building tasks, where both SoFar and VoxPoser score 11.1%, demonstrating that spatio-temporal reasoning and causal memory are critical missing components for reliable long-horizon manipulation. Keywords: Robot Manipulation · Vision-Language Models · Long Horizon Planning · Spatio-Temporal Reasoning · Causal Memory ∗ Equal contribution. † Corresponding author. ♠ Project lead.‡ Work done during an internship at Yuanxing Robotics. 2 Y. Huang et al. 1 Introduction Vision-language models (VLMs) have emerged as powerful planners for robotic manipulation, leveraging internet-scale semantic knowledge and visual perception to translate high-level instructions into executable action sequences [14, 32 , 34 , 81] . VLM-based systems have demonstrated strong performance on short-horizon manipulation tasks, including precise grasping, pose-aware placement, and decomposition of language instructions into subgoal sequences [4, 26 , 47 , 56] . However, this success does not readily extend to long-horizon tasks, which demand sustained spatial and causal awareness across multi-step action sequences that irreversibly alter the environment [55, 66 , 76] . Current VLM-based planners lack the persistent state tracking required to bridge observations across steps, forcing them to reinfer world state from partial observations at each decision step [19, 28 , 79] . Without accumulated context, perceptual errors and action effects compound across steps, ultimately resulting in cascading failures, as illustrated in Fig. 1. In contrast, humans maintain a coherent mental model [35] that continuously integrates spatial relation

FAQs

Why do VLM-based planners struggle with long-horizon robotic manipulation?

They often operate as step-wise observation-to-action mappings, reinferring geometry from raw pixels each decision step and lacking persistent tracking of how actions change the environment. This causes accumulated perceptual errors and cascading precondition violations.

What is RoboStream, and how does it enable persistent geometric grounding?

RoboStream is a training-free framework that uses Spatio-Temporal Fusion Tokens (STF-Tokens) to bind visual evidence to 3D geometric attributes, producing identity-persistent object grounding across steps.

How does RoboStream maintain causal continuity across action sequences?

It maintains a Causal Spatio-Temporal Graph (CSTG) that records action-triggered state transitions across steps, allowing the planner to trace causal chains and preserve object permanence even under occlusion.

Jul 16, 2026

Upload by Finn