[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85157-en":3,"doc-seo-85157-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85157,1374391974468,"Eden","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","Robo-ValueRL Reliable Value Estimation for Offline-to-Online Reinforcement Learning","Offline-to-online reinforcement learning is promising for generalizable robotic manipulation, but its full-stack complexity makes reproduction and diagnosis difficult. Value estimation is central for prioritizing diverse data during policy improvement, yet the link between value-function reliability and optimization remains insufficiently studied. Robo-ValueRL proposes a unified framework for learning history-conditioned value functions, measuring reliability via global-progress and local-preference metrics, and propagating reliable estimates into quality-conditioned offline pretraining and online residual adaptation. Experiments on 240 hours of offline demonstrations and 3,000+ online rollouts show strong performance gains, reaching 86% chip insertion success and 84% block disassembly success.","arXiv :2607 .09866v1 [ cs .RO] 10 Jul 2026  \nRobo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning  \nWenke Xia1 , ∗ , Pei Ren2 , ∗ , Wenbo Yu3 , Yizhuo Zhang1 , Jifan Li1 , Yixue Zhang4 , Yinuo Zhao2 , Qingyang Gao2 , Jianlong Fu5 , Jian Tang2 , Ji-Rong Wen1 , Zhengping Che2 ,B , Di Hu1 ,B  \n1 Gaoling School of Artificial Intelligence, Renmin University of China  \n2 Beijing Innovation Center of Humanoid Robotics  \n3 Beijing Forestry University, 4 Peking University, 5 Microsoft Research  \n∗Equal contribution, B Corresponding author  \nAbstract  \nOffline-to-online reinforcement learning is promising for generalizable robotic manipulation, yet its fullstack complexity obscures reproduction and diagnosis. Within such systems, value estimation plays a central role in prioritizing heterogeneous data for policy improvement. Despite its importance, the central question remains underexplored: how value-function reliability shapes policy optimization in offline-to-online reinforcement learning. To answer this question, we propose Robo-ValueRL, a unified framework that enables reliable value estimation and systematically traces its downstream effects on policy pretraining and online improvement. Concretely, Robo-ValueRL learns a history-conditioned value estimator and evaluatesits reliability through global-progress and local-preference metrics. These resulting value estimates are propagated into quality-conditioned consistency-policy pretraining and a residual adaptation module on online rollouts, providing a unified testbed for analyzing how value reliability shapes downstream policy performance. Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, our extensive experiments show that downstream performance is strongly associated with value reliability. Reliable value functions provide better action-quality estimates, allowing value-guided offline RL to scale more effectively than quality-agnostic behavior cloning, and stabilize online improvement by prioritizing high-quality rollout data. Integrating reliable value guidance through offline pretraining with online improvement, our system achieves 86% success on millimeter-level precise chip insertion and 84% on generalizable block disassembly. We hope these findings highlight the importance of value-guided data utilization for effective policy improvement from heterogeneous robotic experience.  \n[Email:](Email: Wenke Xia at xiawenke2022@ruc.edu.cn)[ Wenke Xia at](Email: Wenke Xia at xiawenke2022@ruc.edu.cn)[ xiawenke2022@ruc.edu.cn](Email: Wenke Xia at xiawenke2022@ruc.edu.cn), [Di Hu at](Di Hu at dihu@ruc.edu.cn)[ dihu@ruc.edu.cn](Di Hu at dihu@ruc.edu.cn),  \n[Pei Ren at](Pei Ren at pei.ren@x-humanoid.com)[ pei.ren@x-humanoid.com](Pei Ren at pei.ren@x-humanoid.com), Zhengping Che [at](at z.che@x-humanoid.com)[ z.che@x-humanoid.com](at z.che@x-humanoid.com)  \nProject Page: [https://gewu-lab.github.io/Robo-ValueRL/](https://gewu-lab.github.io/Robo-ValueRL/)  \nCode: [https://github.com/Open-X-Humanoid/Robo-ValueRL](https://github.com/Open-X-Humanoid/Robo-ValueRL)  \nData&Model: [https://huggingface.co/collections/X-Humanoid/robo-valuerl](https://huggingface.co/collections/X-Humanoid/robo-valuerl)  \n1 Introduction  \nOffline-to-online reinforcement learning has emerged as a promising paradigm for generalizable robotic manipulation [1– 3], bridging offline policy pretraining [4–11] with online improvement through real-world interaction [12–17] . However, unlike conventional benchmarks, it constitutes a full-stack embodied learning system shaped by coupled components, including data curation, value estimation, offline pretraining, and online exploration [18–20] . These interdependencies make performance difficult to reproduce and failures hard to diagnose under deployment conditions.  \nPick PCB Adjust Pick Chip Insert Put  \nGlobel Task-level Progress  \nPick Stack  \nDisassemble Disassemble  \nClassify & Arrange  \nLocal Action-level ","cbCaipZJT13Tm3wp","https://ap.wps.com/l/cbCaipZJT13Tm3wp","pdf",14534618,1,19,"English","en",105,"# Introduction\n# Value Function Estimation\n## Reliability metrics and value-guided learning\n# Robo-ValueRL framework and training pipeline","[{\"question\":\"What problem does Robo-ValueRL address in offline-to-online reinforcement learning?\",\"answer\":\"It targets the difficulty of reproducing and diagnosing full-stack offline-to-online robotic systems, focusing on how value-function reliability impacts policy optimization across offline pretraining and online improvement.\"},{\"question\":\"How does Robo-ValueRL measure value-function reliability?\",\"answer\":\"It uses a history-conditioned value estimator and evaluates reliability with global-progress and local-preference metrics to reflect how well values capture task progress and action quality.\"},{\"question\":\"How does reliable value estimation improve downstream policy performance?\",\"answer\":\"Reliable value functions provide better action-quality estimates, enabling value-guided offline RL to scale over mixed-quality demonstrations and stabilizing online improvement by prioritizing higher-quality rollout data.\"}]",1784201442,48,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"robo-valuerl-reliable-value-estimation-for-offline-to-online-reinforcement-learning","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/robo-valuerl-reliable-value-estimation-for-offline-to-online-reinforcement-learning/85157/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does Robo-ValueRL address in offline-to-online reinforcement learning?","Question",{"text":74,"@type":75},"It targets the difficulty of reproducing and diagnosing full-stack offline-to-online robotic systems, focusing on how value-function reliability impacts policy optimization across offline pretraining and online improvement.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How does Robo-ValueRL measure value-function reliability?",{"text":79,"@type":75},"It uses a history-conditioned value estimator and evaluates reliability with global-progress and local-preference metrics to reflect how well values capture task progress and action quality.",{"name":81,"@type":72,"acceptedAnswer":82},"How does reliable value estimation improve downstream policy performance?",{"text":83,"@type":75},"Reliable value functions provide better action-quality estimates, enabling value-guided offline RL to scale over mixed-quality demonstrations and stabilizing online improvement by prioritizing higher-quality rollout data.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},"General","general"]