[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85684-en":3,"doc-seo-85684-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85684,549758252649,"Ivy","https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819",8,"Research & Report","EgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos","Steerability is a key capability for generalist robot policies, yet it is largely missing in dexterous-hand systems due to insufficient large-scale, language-aligned, action-accurate demonstrations. EgoSteer presents an end-to-end full-stack system that scales dexterous VLA pre-training from egocentric human videos and supports data-efficient real-robot post-training. It combines EgoSmith for high-quality egocentric curation, a unified teleoperation and human-in-the-loop robot stack for DAgger refinement, and EgoSteer, a world-model-enhanced VLA that enables robust free-form instruction following.","arXiv :2607 .09701v1 [ cs .RO] 21 Jun 2026  \nEgoSteer: A Full-Stack System Towards Steerable Dexterous Manipulation from Egocentric Videos  \nYifan Zhong1,2* , Zhang Chen1,2* , Tianrui Guan1,2* , Fanlian Zeng2,3* , Yuyao Ye1,2 , Tianjia He2 , Ka Nam Lui1,2 , Jiayi Li1,2 , Tingrui Zhang1,2 , Ruilin Yan1,2 , Xinhao Ji1,2 , Guangyu Zhao1,2 , Wenjie Lou1,2 , Jiayuan Zhang1,2 , Yuanpei Chen1,2†, Yaodong Yang1,2†  \n1Institute for AI, PKU, 2PKU-PsiBot Joint Lab, 3UPenn.  \nAbstract: Steerability is a defining capability of generalist robot policies, yet remains largely absent in dexterous-hand systems for lack of large-scale, languagealigned, and action-accurate demonstration data. To address this bottleneck, we present a full-stack system that scales dexterous VLA pre-training from egocentric human videos and enables data-efficient real-robot post-training. It integrates EgoSmith, a data pipeline that curates in-the-wild egocentric videos into 9.6K hours of high-quality pre-training data with 9 × higher throughput and better accuracy than prior SOTA; a unified robot stack for teleoperation and human-inthe-loop correction; and EgoSteer, a world-model-enhanced VLA trained on optimized infrastructure. Human-data pre-training equips EgoSteer with languageguided manipulation priors, which are grounded through robot post-training and improved by DAgger refinement. Empirically, EgoSteer robustly executes freeform instructions across 40+ diverse tasks, demonstrating failure recovery, dexterity, and generalization. The pre-trained model also few-shot adapts to complex long-horizon tasks, including box folding, on two embodiments with 75+% success. We open-source the system, data, and model [at](at egosteer.github.io)[ egosteer.github.io](at egosteer.github.io).  \nKeywords: Steerable Dexterous Manipulation, VLA Models, Egocentric Videos  \nFigure 1: Our full-stack system integrates EgoSmith, Robot Stack, and EgoSteer to learn from large-scale egocentric human videos and facilitate data-efficient real-robot post-training, enabling steerable dexterous manipulation across over 40 tasks alongside few-shot adaptation to complex, long-horizon tasks.  \n1 Introduction  \nA central goal of general-purpose embodied intelligence is to enable robots to perform diverse manipulation tasks from open-ended human intent. Despite rapid progress in embodied foundation models [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], most systems still require task-specific finetuning, while the few that follow free-form language are largely limited to grippers [15, 16] . Dual-  \n*Equal contribution.  \n†Corresponding author emails: [yuanpei.chen312@gmail.com and yaodong.yang@pku.edu.cn](yuanpei.chen312@gmail.com and yaodong.yang@pku.edu.cn).  \ndexterous-hand robots provide a more expressive embodiment, with greater actuation capacity and fine-grained interaction potential for general-purpose manipulation. Yet on this more capable but more challenging platform, steerable dexterous manipulation remains largely unrealized.  \nThe key bottleneck lies in data and system scalability. While language-guided manipulation demands large-scale, high-quality data, collecting such demonstrations directly on dexterous robots is exceptionally difficult, particularly for a specific embodiment. Egocentric human videos [17, 18] offer a scalable alternative, as human hand manipulations contain rich interaction knowledge and are spontaneously generated at a massive scale. However, raw egocentric videos are noisy and lack reliable language and action annotations. Without systematic curation, these unstructured videos provide unstable supervision and can degrade downstream robot policies. Even with high-quality human data, the system must employ high-capacity models trained with effective objectives on scalable infrastructure, and ground the learned priors to the target robot. Failing to address any of these co-dependent components prevents the realization of general language-following manipulation.","cbCaiiHdvxHiPod5","https://ap.wps.com/l/cbCaiiHdvxHiPod5","pdf",10162355,1,31,"English","en",105,"# Abstract\n# Introduction\n## Motivation and Data Bottleneck\n## Proposed Full-Stack System (EgoSmith, Robot Stack, EgoSteer)","[{\"question\":\"What problem does EgoSteer address in steerable dexterous manipulation?\",\"answer\":\"EgoSteer targets the lack of large-scale, language-aligned, action-accurate demonstration data and the system scalability issues that prevent steerable dexterous hand policies from emerging reliably.\"},{\"question\":\"How does EgoSmith improve training data quality from egocentric videos?\",\"answer\":\"EgoSmith curates in-the-wild egocentric videos through pre-filtering, 4D motion estimation, language labeling, and post-filtering to produce clean, fully annotated training corpora with higher throughput and more precise annotations.\"},{\"question\":\"What role does the unified robot stack play in the system?\",\"answer\":\"The robot stack enables teleoperation and human-in-the-loop correction by mapping operator relative motions to robot states, supporting efficient DAgger refinement from arbitrary deployment states.\"}]",1784205585,78,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"egosteer-a-full-stack-system-towards-steerable-dexterous-manipulation-from-egocentric-videos","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/egosteer-a-full-stack-system-towards-steerable-dexterous-manipulation-from-egocentric-videos/85684/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does EgoSteer address in steerable dexterous manipulation?","Question",{"text":75,"@type":76},"EgoSteer targets the lack of large-scale, language-aligned, action-accurate demonstration data and the system scalability issues that prevent steerable dexterous hand policies from emerging reliably.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does EgoSmith improve training data quality from egocentric videos?",{"text":80,"@type":76},"EgoSmith curates in-the-wild egocentric videos through pre-filtering, 4D motion estimation, language labeling, and post-filtering to produce clean, fully annotated training corpora with higher throughput and more precise annotations.",{"name":82,"@type":73,"acceptedAnswer":83},"What role does the unified robot stack play in the system?",{"text":84,"@type":76},"The robot stack enables teleoperation and human-in-the-loop correction by mapping operator relative motions to robot states, supporting efficient DAgger refinement from arbitrary deployment states.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]