Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio

Jul 16, 2026

Generative video foundation models show strong compositional priors, yet world-action and video-action models often lose these priors after fine-tuning on robotic action data, creating a video-action generalization gap. This paper evaluates a comprehensive design space of video-action models and finds that standard choices do not yield a clear emergent explanation pattern. It introduces Temporal Ratio (TR), an attention-based metric quantifying reliance on future latent rollouts, which predicts compositional generalization capacity and varies by task phase. TR-Adaptive Guidance mitigates the OOD–ID compositional gap on LIBERO and real-world tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages26

Reading Time1 h 6 min

Introduction

Compositional generalization and the VAG gap

Interface between video foundation models and action prediction

Temporal Ratio (TR) as a diagnostic and predictor

TR-Adaptive Guidance and evaluation

arXiv :2607 .08 127v 1 [ cs .CV] 9 Jul 2026 Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio Utkarsh A. Mishra1* , Yongxin Chen1 , Danfei Xu1 , Yang Liu2 , Xi Chen2 , Jiayuan Mao2 1 Georgia Tech, 2Amazon FAR * Work done during an internship at Amazon FAR Abstract: Generative video foundation models exhibit strong compositional priors, yet world-action models (WAMs) and video-action models (VAMs) often lose these priors after finetuning on robotic action data. We refer to this discrepancy asthe video-action generalization gap. In this paper, we systematically investigate this gap by evaluating a comprehensive design space of VAMs, demonstrating that standard design choices yield no emergent explanation pattern. To explain this behavior, we introduce the Temporal Ratio (TR), an attention-based measure of how strongly the action head relies on future latent rollouts relative to the anchored current frame. TR has two key properties: first, a model’s structural reliance on future-predictive latents, measured via TR, acts as a predictor of its compositional generalization capacity; second, it natively fluctuates based on task phase, shifting attention to future frames during planning and reverting to the present frame for precise manipulation. Finally, based on these findings, we propose an inferencetime adaptive guidance method, which exploits this intrinsic feature attention pattern to dynamically amplify compositional video conditioning signals precisely when the policy relies on future rollouts. Evaluated on the LIBERO benchmark and real-world tasks, our approach mitigates the OOD-ID compositional generalization gap. More details: [https://umishra.me/temporal-ratio/](https://umishra.me/temporal-ratio/) Keywords: Video-Action Models, Compositional Generalization 1 Introduction Compositional generalization [1, 2, 3] is essential for autonomous manipulation: a robot should recombine seen objects, receptacles, and primitives into unseen task sequences. Existing policies often fail under such recompositions because they overfit to visual correlations such as object coordinates [4, 5, 6, 3] . For example, a policy trained on “put the cream cheese in the bowl” and “put the bowl on top of the cabinet” may still fail on “put the cream cheese on top of the cabinet,” despite involving only previously seen objects, receptacles, and primitives. An emerging direction is to use generative video foundation models (VFMs) [7, 8, 9] for action learning, leveraging video models trained on internet-scale video data with strong compositional generalization by capturing temporal coherence, object interactions, and physical plausibility. Consequently, the key challenge is how to transfer these pretrained video priors into robotic control. Current World-Action Models (WAMs) [10, 11] and Video-Action Models (VAMs) [12, 13, 14] typically either (1) predict future video [15, 16](or corresponding latent features [12, 13]) and infer actions through inverse dynamics, or (2) jointly model video and action tokens within a unified transformer [10, 11, 17, 18, 19, 20] . However, despite strong in-domain performance, these methods consistently fail to preserve the compositional generalization abilities of their underlying video backbones (Fig. 1, Top) after finetuning on action data [21] . We refer to this discrepancy as the video-action generalization (VAG) gap. This motivates the following research question: what is the right interface between VFMs and action prediction that preserves compositional priors? In this work, we examine a class of VAMs that use latent features from pre-trained video foundation models and learn a latent inverse-dynamics action head [12, 13, 14] . Such models preserve the video model’s generative prior by feeding latent video features to an action head trained with flow matching [22] . To understand the properties of this interface, we systematically study its design space across

FAQs

What is the video-action generalization (VAG) gap?

The VAG gap is the discrepancy where models trained for robot action prediction lose the compositional priors of the underlying video foundation model after fine-tuning on action data.

How does Temporal Ratio (TR) explain compositional generalization?

TR measures how strongly the action head attends to future latent rollouts relative to the current anchored frame. Higher TR corresponds to stronger compositional generalization capacity and directly predicts the ID–OOD performance gap.

How does TR-Adaptive Guidance mitigate the OOD–ID compositional generalization gap?

TR-Adaptive Guidance uses TR’s intrinsic attention pattern to amplify compositional video conditioning signals during planning phases and relax guidance during precise local manipulation, improving OOD success while preserving ID performance.

Jul 16, 2026

Upload by Theodora