[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86571-en":3,"doc-seo-86571-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":11,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86571,687197207919,"Theodora","https://ap-avatar.wpscdn.com/avatar/a000253d6f5f7c60be?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779446848396160552",8,"Research & Report","Towards Human-level Dexterous Teleoperation","Humans manipulate tools by continuously coordinating in-hand contact transitions that translate, reorient, and regrasp objects within a single hand. Achieving comparable dexterity in robots via teleoperation requires accurate object-motion control driven by dynamic hand–object contacts, yet existing systems remain far from this standard. TELEDEXTER introduces a hand–object co-tracking controller that converts operator intent into learned low-level contact execution using consecutive co-tracking subgoals and a hybrid sparse–dense reward. Single-stage RL with random action masking enables zero-shot sim-to-real transfer to a real robot, validated on seven tasks with a 75% average success rate where baselines fail.","arXiv :2607 . 11481v1 [ cs .RO] 13 Jul 2026  \nTowards Human-level Dexterous Teleoperation  \nPuhao Li 1 ,2 ,∗ , Zeyuan Chen2 ,3 ,∗ , Yingying Wu 1 ,2 ,∗ , Pengkun Wei2 , Yuyang Li2 ,3 , Tianyu Wang2 ,3 , Jiaxiao Shi2 , Mingrui Yu 1 , Baoxiong Jia2 , Song-Chun Zhu 1 ,2 ,3 , Tengyu Liu2 ,†, Siyuan Huang2 ,†  \n1Tsinghua University 2 State Key Lab of General Artificial Intelligence, BIGAI  \n3Peking University ∗Equal Contribution †Corresponding author  \n[https://bigai-dex.github.io/blog/teledexter](https://bigai-dex.github.io/blog/teledexter)  \nteleoperated  \nFig. 1: TELEDEXTER learns diverse dexterous in-hand manipulation skills within a single-stage framework, marking a concrete step towards human-level dexterous teleoperation.  \nAbstract: Humans routinely wield tools, swap grasps, and reposition objects within a single hand—seamlessly orchestrating contact transitions that span translation, re-orientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand–object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TELEDEXTER, a hand-object co-tracking controller that maps operator intent into learned, lowlevel contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation.  \nThe entire pipeline requires only single-stage RL and, with random action mask-  \ning and domain randomization, transfers zero-shot to the real robot. We evaluate TELEDEXTER on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.  \nKeywords: Robot Manipulation, Dexterous Teleoperation, Sim-to-Real Transfer  \n1 Introduction  \nEveryday manipulation demands human-level dexterity: the ability to dynamically reorient, translate, and regrasp objects within a single hand by continuously coordinating complex finger–object contacts [1, 2] . While such agility is effortless for humans, it remains far beyond the reach of current robots. Teleoperation offers a powerful paradigm for closing this gap by enabling operators to directly teach the robot in the loop [3–5] . However, achieving human-level in-hand dexterity through dexterous teleoperation remains an open challenge.  \nDespite recent progress, existing dexterous teleoperation systems still fall short of human-level inhand dexterity. One dominant line of work employs kinematic retargeting to map human hand motion directly onto robotic topologies, leveraging vision-based tracking [5–8] or wearable exoskeleton gloves [9–12] . While this paradigm provides an intuitive, low-latency interface that faithfully captures the operator’s kinematic intent, it completely ignores hand-object contact forces and object inertia. Consequently, high-acceleration maneuvers, non-prehensile interactions, and continuous finger-gaiting remain highly unstable, frequently resulting in object slippage or drop failures.  \nAn alternative paradigm [13] learns a dexterous action prior in simulation to map coarse teleoperation commands to contact-rich hand actions, improving local contact robustness over pure kinematics. However, these methods typically rely on synthetically generated grasp transitions as training targets, which often lack physical feasibility. Furthermore, encoding the action prior into a generative model introduces cascadin","cbCaic8UPg7sbxmg","https://ap.wps.com/l/cbCaic8UPg7sbxmg","pdf",17811232,1,27,"English","en",105,"# Introduction\n## Limitations of existing dexterous teleoperation\n## TELEDEXTER: hand–object co-tracking controller\n## Key technical designs\n### Consecutive subgoal co-tracking\n### Co-tracking sequence construction\n### Sim-to-real robustness","[{\"question\":\"Why is human-level in-hand dexterity difficult for current robots in teleoperation?\",\"answer\":\"Current systems do not reliably handle dynamic hand–object contact forces, object inertia, and stable finger-gaiting during high-acceleration or non-prehensile interactions, often causing slippage or drops.\"},{\"question\":\"What does TELEDEXTER change compared with kinematic retargeting approaches?\",\"answer\":\"Instead of mapping hand joints alone, TELEDEXTER uses explicit synchronized geometric targets for both fingertip positions and object pose, while a low-level RL controller realizes the dual co-tracking goals under multi-contact physics.\"},{\"question\":\"How does TELEDEXTER enable zero-shot transfer from simulation to a real robot?\",\"answer\":\"The training uses a single-stage RL pipeline together with random action masking and domain randomization, improving sim-to-real robustness during real-world execution.\"}]",1784212701,68,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"towards-human-level-dexterous-teleoperation","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/towards-human-level-dexterous-teleoperation/86571/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-28","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":11},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is human-level in-hand dexterity difficult for current robots in teleoperation?","Question",{"text":75,"@type":76},"Current systems do not reliably handle dynamic hand–object contact forces, object inertia, and stable finger-gaiting during high-acceleration or non-prehensile interactions, often causing slippage or drops.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What does TELEDEXTER change compared with kinematic retargeting approaches?",{"text":80,"@type":76},"Instead of mapping hand joints alone, TELEDEXTER uses explicit synchronized geometric targets for both fingertip positions and object pose, while a low-level RL controller realizes the dual co-tracking goals under multi-contact physics.",{"name":82,"@type":73,"acceptedAnswer":83},"How does TELEDEXTER enable zero-shot transfer from simulation to a real robot?",{"text":84,"@type":76},"The training uses a single-stage RL pipeline together with random action masking and domain randomization, improving sim-to-real robustness during real-world execution.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]