WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos

Jul 16, 2026

Generalizable robot policies usually depend on action-labeled demonstrations, but such data are costly and hard to scale, while human videos are abundant yet lack action annotations for direct robot control. WALA presents a framework that jointly learns executable latent actions from both action-labeled demonstrations and action-free videos. It pretrains a semantic-geometric latent action model from videos by learning representations of scene evolution using sparse future observations and predicts future deltas in DINOv3 and dense depth spaces. Policy training freezes the encoder and trains the decoder as a latent world model, with supervision from robot action prediction, latent target matching, and future dynamics prediction. Experiments show strong results on RoboTwin and 75.2% average success on RoboCasa, with additional real-robot validation of generalization across manipulation tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Problem Motivation

Limitations of Existing VLA Training

Role of Action-Free Human Videos

World Models and Future Dynamics as Training Signals

WALA Learning Executable Latent Actions from Action-Labeled Demonstrations and Action-Free Videos Jiahao Liu 1,2,3 , Zhongpu Xia2,†, Shuai Tian1,3 , Huangrui Li 1,2,3 , Yuhang Zheng4 , Ning Ma2,5 , Xin Fu2,3 , Xiaotian Liu2 , Jing Li2 , Yixian Li2 , ShangQing Zhou 1,2,3 , Zebin Xing 1,3 , Linbo Wang 1,3 , Chaoyue Li 1,3 , Haoran Li 1,3,∗ , Dongbin Zhao 1,3,∗ arXiv :2607 . 11397v1 [ cs .RO] 13 Jul 2026 Abstract—Generalizable robot policies typically rely on robot demonstrations with action annotations, yet such data are expensive to collect and difficult to scale. In contrast, largescale and readily available human videos record rich physical interactions, but lack action annotations that can be directly used for robot control. We present WALA, a framework that jointly learns executable latent actions from action-labeled demonstrations and action-free videos. WALA first pretrains a semantic-geometric latent action model on videos without action annotations, enabling it to learn action-relevant representations from scene evolution between the current observation and multiple sparsely sampled future observations. Specifically, WALA forms semantic and geometric future deltas, from which the encoder extracts latent action targets, while the decoder predicts future deltas in the DINOv3 feature space and dense depth space. This avoids raw pixel reconstruction, reducing the influence of appearance details while preserving task-relevant semantic and geometric structure. During policy training, the pretrained encoder remains frozen to providestable latent action targets, while the decoder serves as a trainable latent world model. The latent actions generated by the vision-language backbone are jointly supervised by robot action prediction, latent action target matching, and future dynamics prediction. Action-labeled demonstrations provide both executable control and dynamics supervision, whereas actionfree videos require no robot action labels and still participate in training through latent action and future dynamics supervision. In this way, WALA connects physical scene evolution in videos with executable robot control. Experiments show that WALA achieves strong performance on RoboTwin and reaches an average success rate of 75.2% on RoboCasa, setting a new stateof-the-art result. Additional real-robot experiments further evaluate its generalization ability across diverse manipulation tasks. Project page: WALA Project Page I. INTRODUCTION Learning generalizable robot policies requires data and supervision that cover diverse objects, scenes, task compositions, robot embodiments, and physical interactions. In recent years, vision-language-action models (VLAs) have made significant progress on language-conditioned manipulation by transferring large-scale vision-language pretraining to robot control [7], [8], [9], [10],[11] . Despite this progress, current VLAs still face two coupled limitations. First, they mainly rely on robot demonstrations with action annotations, which ∗ Corresponding author. †Project leader. 1CASIA, 2Anyverse Dynamics, 3UCAS, 4NUS, 5XJYLU. are expensive to collect, difficult to align across platforms, and insufficient to cover the long tail of physical interactions in the real world. Second, the standard action-supervised training objective primarily learns a mapping from current observations and language instructions to robot actions. It provides only weak and indirect supervision about the future physical consequences of those actions. For long-horizon, contact-rich, or spatially precise manipulation tasks, a policy must not only output plausible motor commands, but also understand how objects, contacts, and scene geometry should evolve after an action is executed. In contrast, large-scale and readily available human videos record rich physical interactions at much lower cost, including object pose changes, contact events, occlusion relationships, tool use, and goal-directed behavior. These

FAQs

为什么WALA要同时利用动作标注演示和无动作标注视频？

动作标注演示能提供可执行控制与动力学监督，但收集成本高；无动作视频包含丰富的物理世界演化信息，却缺少可用于低层监督的动作标签。WALA通过把无动作视频也纳入潜在动作与未来动力学监督来补足两者的不足。

WALA的核心训练目标如何让潜在动作可执行？

WALA用视觉-语言骨干生成潜在动作，并通过机器人动作预测、潜在动作目标匹配以及未来动力学预测对潜在动作进行联合监督，使其同时对控制与未来场景演化产生对齐。

语义-几何潜在动作模型如何从视频中学习？

WALA在无动作标注视频上预训练语义-几何潜在动作模型：它在当前观测与多个稀疏采样的未来观测之间形成语义与几何的future deltas，然后编码器提取与动作相关的潜在目标，并在DINOv3特征空间与稠密深度空间预测未来deltas，以避免直接像素重建的干扰。

Jul 16, 2026

Upload by Genevieve