WAM-TTT: Steering World Action Models by Watching Human Play at Test Time

Jul 16, 2026

Upload by Ezra

WAM-TTT presents a test-time training approach for steering robot foundation models toward new task variants using raw human videos, without additional robot demonstrations or task-specific fine-tuning. Instead of imitating trajectories, human videos are absorbed into a lightweight adaptive memory inside a frozen World Action Model via self-supervised video prediction. A meta-training stage aligns human demonstrations with robot behaviors using paired human-robot data and a key–value memory reconstruction objective. Deployment adapts memory from unlabeled human videos only, preserving foundation generalization while improving performance across manipulation tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages28

Reading Time1 h 11 min

Introduction

Motivation for test-time steering

WAM-TTT framework overview

Contributions

WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time arXiv :2607 .06988v2 [ cs .RO] 10 Jul 2026 Yusen Feng 1 ,2 ,∗ Bingchen Han 1 ,2 ,∗ Jiangran Lyu 1 ,2 ,∗ Kai Liu2 ,3 Yixin Zheng2 ,3 Yuxuan Wan 1 ,2 Weiheng Liu2 ,3 Sun Han 1 ,2 Ruiqin Li 1 ,2 Yulong Zhang 1 Fangfu Liu4 Xuesong Shi2 Libin Liu 1 ,† Yizhou Wang 1 ,† Zhizheng Zhang2 ,† He Wang 1 ,2 ,† 1Peking University 2 Galbot 3 CASIA 4Tsinghua University ∗Equal contribution †Corresponding authors Office ••• No retargeting: Unlabeled Human videos to guide robot actions directly Human Demonstrations Train Figure 1: Overview of WAM-TTT. Given unlabeled human demonstrations from diverse environments, WAM-TTT steers a pretrained World Action Model (WAM) without retargeting, robot actions, or human-side annotations. During deployment, human videos are absorbed into lightweight TTT fast weights through self-supervised video prediction, while the pretrained action model remains frozen. The adapted memory then guides robot execution through the WAM’s shared visual-action dynamics, enabling efficient and reusable steering from human demonstrations. Abstract: Steering robot foundation models (RFMs) toward new task variants or user-preferred behaviors remains challenging, often requiring additional robot demonstrations, task-specific fine-tuning, or long-context conditioning. We present WAM-TTT, a test-time training framework for steering world action models from raw human videos. Rather than treating human videos as trajectories to imitate, WAM-TTT absorbs them into a lightweight adaptive memory inside a frozen WAM through self-supervised video prediction. To make this memory useful for control, we introduce a meta-training stage that aligns human demonstrations with robot behaviors using paired human-robot data and a key–value memory reconstruction objective. At test time, only unlabeled human videos are required to adapt the memory, while the pretrained WAM remains frozen. This enables efficient and reusable steering without robot actions, human-side annotations, or task-specific fine-tuning, while preserving the generalization ability of the foundation model. Extensive experiments show that WAM-TTT consistently outperformsin-context human-video conditioning baselines across diverse manipulation tasks and generalization settings. Keywords: World Action Model, Test-time Training, Human Videos 1 Introduction Recently, the robotics community has increasingly pursued general-purpose robot foundation models through large-scale pretraining. However, most existing RFMs primarily absorb knowledge into fixed model parameters. Once deployed, their behavior is largely determined by the pretrained weights and a limited conditioning interface, such as language instructions, goal images, or short observation histories[1, 2, 3, 4, 5, 6] . As a result, steering RFMs toward new task variants, object interactions, or user-preferred strategies typically requires collecting additional robot demonstrations or fine-tuning the full model. This limits the flexibility and reusability of RFMs in open-ended deployment settings, where users may wish to quickly specify new behaviors without retraining a robot policy. Human demonstrations offer a natural and scalable interface for steering RFMs[7, 8, 9, 10, 11, 12]: users can simply show how objects should be handled, without specifying robot actions. Existing methods typically leverage human videos through co-training or fine-tuning with robot data [7, 8, 9, 10, 13, 11, 12], often relying on additional supervision such as hand poses, 3D motion, or retargeted trajectories [2, 14, 3, 15, 4, 6, 16, 17] . Such supervision can be noisy and costly to obtain, while taskspecific fine-tuning may cause catastrophic forgetting and reduce the reusability of the pretrained model. A more direct alternative is to condition robot policies on raw human videos [18], but this requires learning such capabilities during large-scale

FAQs

What problem does WAM-TTT address in robot foundation models?

Steering robot foundation models to new task variants or user-preferred behaviors is difficult because most methods require additional robot demonstrations, task-specific fine-tuning, or long-context conditioning.

How does WAM-TTT use human videos differently from imitation-based approaches?

WAM-TTT treats human videos as deployment-time memory. The framework absorbs videos into lightweight adaptive memory inside a frozen World Action Model using self-supervised video prediction instead of learning direct trajectory imitation.

What is updated at test time, and what remains frozen?

At deployment, only unlabeled human videos are used to update the adaptive memory through video prediction, while the pretrained WAM remains frozen. This enables efficient steering without robot actions or human-side annotations.

Jul 16, 2026

Upload by Ezra