Smooth Operator: A Real-Time Sampling-Based Algorithm for Kinematic Hand Retargeting

Jul 16, 2026

Advances in learning-based robotic manipulation, such as VisionLanguage-Action (VLA) models and Video Action Models (VAMs), depend on high-quality teleoperation data. That capability is constrained by the fidelity of human demonstrations, and gradient-based retargeting often converges to different local minima, producing jitter that harms data quality and operator experience. This work introduces the Sampling-Based Retargeter (SBR), a gradient-free, low-jitter real-time method, evaluated in simulation and a real-world user study with 18 participants on three tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

Introduction

Problem with gradient-based retargeting

Evaluation challenges and experimental design

Main contribution: Sampling-Based Retargeter (SBR)

Smooth Operator: A Real-Time Sampling-Based Algorithm for Kinematic Hand Retargeting Robert Jomar Malate 1 ,3 ,∗ Erik Bauer2 Norica Bacuieti3 Stefanos Charalambous3 Elvis Nava3 Robert K. Katzschmann 1 ,3 ,∗ Benedek Forrai3 ,∗ 1ETH Zurich, Switzerland 2 Stanford University, United States 3mimic robotics, Switzerland arXiv :2607 .0749 1v 1 [ cs .RO] 8 Jul 2026 * Corresponding authors: [rmalate@student.ethz.ch](rmalate@student.ethz.ch), [rkk@ethz.ch](rkk@ethz.ch), [and](and benedek.forrai@mimicrobotics.com)[ benedek.forrai@mimicrobotics.com](and benedek.forrai@mimicrobotics.com) [https://mimicrobotics.github.io/smooth-operator/](https://mimicrobotics.github.io/smooth-operator/) Figure 1: Overview. Drawing from the rich traditions of sampling-based control in robotics, we introduce a sampling-based retargeter for tracking human hand pose commands with dexterous robotic hands (a) . We rigorously evaluate its performance against the state of the art along several metrics, both in a simulated environment (b) that allows quick iteration, and in a real-world user study involving 18 participants on 3 complex manipulation tasks (c) . We find that our novel sampling based method leads to a better user experience, reduced jitter, and achieves state of the art performance in task success rates and operator workload scores. Abstract: Advances in learning-based robotic manipulation, such as VisionLanguage-Action (VLA) models and Video Action Models (VAMs), heavily rely on high-quality teleoperation data. Their capabilities are strictly upper-bounded by the quality of the underlying human demonstrations. Current gradient-based retargeting algorithms often converge to different local minima, resulting in jitter that affects data quality and teleoperation experience. To address this, we introduce the Sampling-Based Retargeter (SBR), a novel gradient-free retargeting method drawn from the rich literature of sampling-based control and explicitly designed for low-jitter, real-time kinematic retargeting. We evaluate SBR both in simulation and through a rigorous real-world user study involving 18 participants performing 3 complex manipulation tasks. Compared to gradient-based baselines, SBR achieved the highest overall task success rate (54.1%) while significantly reducing operator cognitive fatigue, recording the lowest NASA-TLX workload score (36.4 out of 100) . Ultimately, we establish SBR as a highly effective, intuitive retargeter for dexterous manipulation, providing the community with a rigorous benchmarking methodology to guide future retargeting research. Keywords: Teleoperation, Dexterous Manipulation, Retargeting, Sampling Based Control 1 Introduction Retargeting algorithms enable humans to bridge the human-robot embodiment gap, providing a method to control and teach robots to perform dexterous behaviors autonomously. Defined as the mapping of intent from a source to a target [1], retargeting is a critical component of the robot learning data flywheel, providing a mechanism for extracting human manipulation data through real-time control [2, 3, 4] or offline processing [5, 6] . We call the former and latter online and offline retargeting, respectively. However, retargeting is challenging due to the complex and nonlinear mapping between the human and robot hand manifolds [7] . This complexity arises from the kinematic discrepancies between the embodiment gap. To address the embodiment gap, a wide variety of approaches have been developed to translate human hand motion into executable robot commands [8] . These retargeting approaches commonly use gradient-based optimization, which relies on local derivative information to map human intent to robot joints. Gradient solvers, however, risk converging into undesirable local minima, thereby generating aggressive, high-frequency command spikes (jitter) . In real-world teleoperation, this erratic behavior degrades grasp stability and makes aggressive low-pass filtering ne

FAQs

What benefit does smoother retargeting provide for operators?

Smoother retargeting reduces jitter and operator cognitive fatigue, improving task success while lowering workload as measured by NASA-TLX.

Jul 16, 2026

Upload by Seraphina