GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency

Jul 16, 2026

Upload by Quinn

Generated videos provide visual motion priors for robot manipulation, but visual plausibility cannot guarantee physical executability. Many generated sequences lack metric geometry, grasp grounding, kinematic feasibility, and execution-time feedback, so direct trajectory replay is unreliable in real settings. GenVid2Robot introduces a rigid-geometric consistency framework that validates generated 2D motion using sparse relative SE(3) grounding from an initial RGB-D frame, transfers only geometrically consistent motion to the robot, and applies grasp-conditioned execution. A bounded depth-compensation module mitigates RGB-D noise and contact-induced displacement.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages10

Reading Time25 min

Introduction

Motivation: visual motion priors vs physical executability

Gap: semantic, geometric, and grasp feasibility mismatch

Approach: rigid-geometric consistency as intermediate validation

GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency Haohui Huang, Member, IEEE, Xi Yuan, Panpan Liao, Tao Teng, Member, IEEE, Chenguang Yang, Fellow, IEEE, Jing Guo, Member, IEEE, and Yi Guo, Member, IEEE arXiv :2607 .09191v1 [ cs .RO] 10 Jul 2026 Abstract—Generated videos provide useful visual motion priors for robot manipulation, but their visual plausibility does not imply physical executability. A generated video usually lacks metric geometry, grasp grounding, robot kinematic feasibility, and execution-time feedback, which makes direct trajectory replay unreliable in real-world manipulation. This paper presents GenVid2Robot, a rigid-geometric consistency framework that converts generated video motion into executable real-robot manipulation trajectories. Given an initial RGB-D observation anda task instruction, GenVid2Robot samples task-relevant semantic anchors from the real first frame, tracks these anchors through generated video candidates, and verifies whether the resulting 2D motion can be explained by first-frame RGB-D anchors under a sparse relative SE(3) model. In this way, generated videos are treated as uncertain visual motion hypotheses rather than direct robot demonstrations. Only geometrically consistent motion is transferred to the robot. The accepted relative motion is then applied to the real grasp-time TCP pose selected by maskconstrained grasping, producing a grasp-conditioned execution trajectory that is consistent with both the visual motion prior and the physical grasp configuration. To reduce execution mismatch caused by RGB-D noise, calibration residuals, and small contact-induced displacement, a bounded depth-compensation module corrects local depth-direction errors without assuming full online replanning. Real-robot experiments demonstrate that GenVid2Robot improves the reliability of generated-video-guided manipulation by grounding visual motion priors with sparse metric geometry, grasp constraints, robot feasibility checking, and bounded execution feedback. Index Terms—Robot manipulation, video generation, visual motion priors, rigid-geometric consistency, sparse 6D motion recovery, grasp-conditioned trajectory induction, depth-corrected execution. I. INTRODUCTION Robots are increasingly expected to perform manipulation tasks from visual observations and high-level language instructions, rather than relying on manually designed trajectories for every object, scene, and task variation. Tasks such as pouring water, lifting a lid, delivering a tool, or sweeping an object require the robot to infer task-relevant object motion, establish a feasible grasp, and execute a physically valid trajectory under sensing and calibration uncertainty. Although reinforcement learning, imitation learning, and diffusion-based visuomotor Corresponding author: Tao Teng. Haohui Huang, Xi Yuan, Panpan Liao, and Jing Guo are with the School of Automation, Guangdong University of Technology, Guangzhou, China. Tao Teng is with the University of Liverpool, Liverpool, U.K. (e-mail: [Tao.Teng@liverpool.ac.uk](Tao.Teng@liverpool.ac.uk)). Chenguang Yang is with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China. Yi Guo is with the State Key Laboratory of Submarine Geoscience, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China. policies have achieved strong results in robot manipulation, their performance is still closely tied to the availability, diversity, and embodiment compatibility of physical demonstrations or robot interaction data [1]–[3] . Foundation models provide a promising way to reduce task-specific supervision by introducing language, semantic, and spatial priors into robot manipulation. Large multimodal models can generate plans, reason about affordances, produce executable code, or define spatial constraints for manipulation [4]–[9] . However, these high-level prior

FAQs

Why can’t generated videos be directly retargeted to robot trajectories?

Generated videos provide image-space dynamics but real execution requires metric 3D geometry, rigid-body consistency, grasp feasibility, and execution-time correction. Visual plausibility can hide drifting correspondences, inconsistent depth, or motions that the robot cannot realize.

How does GenVid2Robot convert video motion into executable robot manipulation?

Given an initial RGB-D observation and a task instruction, it samples task-relevant semantic anchors from the first frame, tracks them through generated video candidates, and verifies that the resulting 2D motion is explainable under a sparse relative SE(3) model grounded by the first-frame RGB-D anchors. Only geometrically consistent motion is transferred.

What mechanisms improve robustness during real-world execution?

GenVid2Robot applies the accepted relative motion to a grasp-conditioned TCP pose chosen by mask-constrained grasping, then uses a bounded depth-compensation module to correct local depth-direction errors caused by RGB-D noise, calibration residuals, and small contact-induced displacement without full online replanning.

Jul 16, 2026

Upload by Quinn