DemoBridge A Simulation in the Loop Toolkit for Single View Human Demonstration Retargeting

Jul 16, 2026

Upload by Aurelia

DemoBridge is a toolkit that converts a single-view RGB stereo recording of a human hand demonstration into an executable robot-arm trajectory validated by physics. Retargeting is difficult across the embodiment gap because inverse kinematics may lack collision-free solutions and feasibility must hold at every waypoint. Single-view perception adds jitter, drift, and occlusion near contact. DemoBridge uses a collision-aware whole-trajectory optimizer with a physics-in-the-loop simulator to validate each phase, backtrack on failure, and re-plan. It also supports simulation rollouts for policy learning with automatic grasp timing and swappable pipeline modules.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages8

Reading Time20 min

Introduction

Problem and challenges

Related systems and approach

Contributions

Related Work

DemoBridge: A Simulation-in-the-Loop Toolkit for Single-View Human Demonstration Retargeting Zehao Wang§ , Fabien Despinoy†, Sergey Zakharov‡, Tinne Tuytelaars§ , and Rahaf Aljundi† § KU Leuven †Toyota Motor Europe ‡Toyota Research Institute arXiv :2607 .09519v1 [ cs .RO] 10 Jul 2026 Abstract—We present DemoBridge, an toolkit that turns a single-view RGB stereo recording of a human hand demonstration into an executable, physics-validated robot-arm trajectory. Retargeting across the embodiment gap is hard. A robot arm reaches a target with a long, articulated body whose links carry far more collision volume than a hand. Solving inverse kinematics (IK) for the mapped end-effector pose often yields no collisionfree solution, and a trajectory imposes this at every waypoint. A single view adds noise, leaving the demonstrated reference inaccurate. At the core of DemoBridge is a single collision-aware planner. It optimizes the whole joint trajectory at once, reasoning jointly over alternative grasp poses, whole-arm and graspedobject collision, and fidelity to the demonstrated path. A physics simulator runs in the loop. It validates each phase as it is produced and backtracks on failure, so a demonstration that cannot be reproduced as given is re-planned rather than discarded. The resulting action sequence is dynamically stable and faithful to the demonstrated manipulation. It also doubles as a ready-to-use simulation rollout for policy learning. Grasp timing is inferred automatically, and the perception backends, robot, and pipeline stages are swappable from configuration. We evaluate wholepipeline retargeting on three real-demonstration tasks and the planner on a controlled synthetic benchmark. Our code is available [at](at gitlab.kuleuven.be/u0123974/demo-bridge)[ gitlab.kuleuven.be/u0123974/demo-bridge](at gitlab.kuleuven.be/u0123974/demo-bridge). I. INTRODUCTION Learning robot manipulation policies from demonstrations is effective, but collecting such demonstrations at scale remains difficult. Teleoperation [14] collects demonstrations by directly controlling the robot, which requires exclusive access to the platform and can interrupt ongoing deployment, such as on an assembly line. Motion capture, whether multi-camera rigs [18, 19] or instrumented gloves [8], needs a calibrated setup or worn sensors. Both confine demonstrations to a controlled environment. Ordinary single-view human video needs only one camera. A tool that turns such a recording into robot motion would let manipulation data be gathered passively and in place. It would also feed the real-to-sim-to-real and imitation methods that learn from human video [1, 2, 9, 24] . The recorded hand motion must be transcribed into a trajectory a specific robot can physically execute, across the embodiment gap between a human hand and a robot arm. We provide the toolkit that performs this transcription, turning a single-view human recording into a robot-executable, physics-validated trajectory. Two difficulties stand in the way, and they are coupled. The first is kinematic. A robot arm spans the workspace with an articulated chain of large collision volume. Even a single mapped end-effector target may have no collisionfree inverse-kinematics (IK) solution. Retargeting a trajectory further compounds the problem, since such feasibility must be maintained at every waypoint along the demonstrated path. The second is perceptual. From a single view, hand keypoints and object poses jitter, drift, and are occluded at the moment of contact. The demonstrated reference is therefore inaccurate exactly where it matters most. Feasibility, contact, and reference fidelity jointly determine an executable motion, so they must be addressed together. The systems closest to ours reach the demonstrated poses one at a time, by per-frame IK or by interpolating sparse keyframes [26, 27] . This is local: it reproduces each pose but reasons neither about whole-arm collision along the pat

FAQs

What does DemoBridge convert, and what output does it generate?

DemoBridge converts a single-view human hand demonstration into a robot-executable trajectory for a robot arm. The produced trajectory is validated by a physics simulator.

Why is retargeting from a human hand to a robot arm difficult?

The embodiment gap makes kinematic feasibility hard: a robot’s articulated links create collision volume, so inverse kinematics for an end-effector target may be collision-free impossible. Perception from a single view also degrades the demonstrated reference with noise and occlusion, especially around contact.

How does DemoBridge ensure a trajectory remains feasible during execution?

It uses a single collision-aware planner that optimizes the whole joint trajectory while tracking the demonstrated path and avoiding collisions for both the arm and the grasped object. A physics simulator runs in the loop and triggers re-planning if any phase fails, rather than discarding the demonstration.

Jul 16, 2026

Upload by Aurelia