PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulation

Jul 16, 2026

Video-based manipulation extracts object-centric motion priors from human demonstrations, generated videos, or RGB-D observations, yet these priors are embodiment-agnostic and cannot be executed directly by a specific robot. PhysV2A introduces a reachability-gated and semantic-mask-constrained feasibility-completion framework that converts video-derived 6D object motion into robot-executable manipulation trajectories. Grasp feasibility is treated as trajectory-conditioned by coupling each 6-DoF grasp candidate with recovered object motion, then selecting feasible candidates via hierarchical robot kinematic checks and ranking by execution suitability. A VLM-assisted, rule-validated S-Mask preserves task-critical Cartesian components while refining manipulability through redundancy-first optimization and bounded Cartesian relaxation, improving success and reducing feasibility failures in real-robot tabletop tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

Introduction

Motivation and challenges

Proposed PhysV2A framework

PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulation Haohui Huang, Junda Duan, Tao Teng, and Chenguang Yang arXiv :2607 .09365v1 [ cs .RO] 10 Jul 2026 Abstract—Video-based manipulation provides object-centric motion priors from human demonstrations, generated videos, or RGB-D observations, but such priors are typically embodimentagnostic and cannot be directly executed by a specific robot. This paper presents PhysV2A, a reachability-gated and semanticmask-constrained feasibility-completion framework for converting video-derived 6D object motion into robot-executable manipulation trajectories. The key idea is to treat grasp feasibility as trajectory-conditioned rather than local: each RGB-D-generated 6-DoF grasp candidate is rigidly coupled with the recovered object motion to form a grasp-conditioned TCP trajectory hypothesis. PhysV2A then performs hierarchical reachabilitygated selection, where infeasible grasp–trajectory pairs are rejected by robot-centric kinematic checks and surviving candidates are ranked by downstream execution suitability. For the selected reachable trajectory, a VLM-assisted and rulevalidated S-Mask identifies task-critical and relaxable Cartesian components, enabling semantic-mask-constrained manipulability refinement through redundancy-first optimization and bounded Cartesian relaxation. Real-robot experiments on four tabletop manipulation tasks show that PhysV2A improves task success over representative video-prior and IK-only baselines, reduces kinematic-feasibility failures, and produces better-conditioned trajectories with bounded semantic deviations. Index Terms—Robot manipulation, video-to-robot, visual motion prior, semantic mask, reachability, manipulability, inverse kinematics, kinematic feasibility. I. INTRODUCTION RECENT advances in video generation, RGB-D tracking, vision-language reasoning, and robot learning have made visual observations an increasingly useful source of tasklevel priors for robot manipulation. Human demonstrations, generated videos, and RGB-D observations can describe how objects should move during manipulation, while point tracking and 6D pose recovery provide geometric interfaces for converting image-space motion into object-space trajectories [1]–[3] . Video-conditioned manipulation methods further exploit observed or generated videos as object-centric demonstrations, where the visual sequence specifies task intent without directly specifying robot joint commands [4]–[7] . These developments motivate a video-to-robot manipulation pipeline that recovers object motion from visual priors and retargets it to a specific robot embodiment. However, visual motion plausibility does not directly imply robot-specific executability. A video-derived object trajectory Haohui Huang and Junda Duan are with the School of Automation, Guangdong University of Technology, Guangzhou, China (e-mail: [hh.huang@ieee.org](hh.huang@ieee.org); [2112404293@mail2.gdut.edu.cn](2112404293@mail2.gdut.edu.cn)). Tao Teng is with the University of Liverpool, Liverpool, U.K. (e-mail: [Tao.Teng@liverpool.ac.uk](Tao.Teng@liverpool.ac.uk)). Chenguang Yang is with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China (e-mail: [cyang@ieee.org](cyang@ieee.org)). Corresponding author: Chenguang Yang. Fig. 1. Motivation. Video-derived motion priors can encode task intent, but visually plausible grasps and trajectories may still violate reachability, fulltrajectory IK, endpoint constraints, or manipulability requirements. may be coherent in image or object space, but the TCP trajectory induced on a specific manipulator can still violate workspace limits, joint limits, IK continuity, or singularityrelated constraints. More importantly, grasp feasibility in video-to-robot manipulation is not a purely local property of a grasp pose. A high-confidence 6-DoF grasp candidate generated from RGB-D obs

FAQs

What problem does PhysV2A address in video-to-robot manipulation?

PhysV2A addresses the gap between visually plausible, embodiment-agnostic motion priors and robot-specific executability. It focuses on converting video-derived 6D object motion into trajectories that satisfy reachability and feasibility constraints on a given manipulator.

How does PhysV2A model grasp feasibility?

PhysV2A treats grasp feasibility as trajectory-conditioned rather than purely local grasp confidence. Each RGB-D-generated 6-DoF grasp candidate is rigidly coupled with recovered object motion to form a grasp-conditioned TCP trajectory hypothesis, which is then evaluated end-to-end.

How does PhysV2A refine a reachable trajectory without damaging task-critical motion?

PhysV2A uses a VLM-assisted and rule-validated S-Mask to identify task-critical versus relaxable Cartesian components. Manipulability is improved via redundancy-first optimization while applying bounded Cartesian relaxation so semantic deviations remain controlled.

Jul 16, 2026

Upload by Clementine