[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82094-en":3,"doc-seo-82094-105":29,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82094,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",8,"Research & Report","SafeExplorer An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions","SafeExplorer trains reinforcement-learning agents on real robots while minimizing costly falls during learning. Standard recovery-policy approaches reduce damage but bias on-policy PPO updates because rollouts come from a mixed behavior policy, and importance-sampling corrections fail when recovery is deterministic. SafeExplorer introduces a drop-in PPO modification with an unbiased policy-gradient estimator that uses score-function terms only at safe timesteps, remaining valid for deterministic recovery. Additional mechanisms accelerate learning near safe boundaries and achieve large reductions in training-time falls while matching or improving final reward.","arXiv :2607 .08925v 1 [ cs .LG] 9 Jul 2026  \nSafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions  \nElham Daneshmand [elham. daneshmand@mail. mcgill. ca](elham. daneshmand@mail. mcgill. ca)  \n[McGill University & Mila](McGill University & Mila)  \nMajid Khadiv [majid.khadiv@tum. de](majid.khadiv@tum. de)  \nTechnical University of Munich  \n[Glen Berseth](Glen Berseth glen. berseth@umontreal. ca)[ glen. berseth@umontreal. ca](Glen Berseth glen. berseth@umontreal. ca)  \nUniversité de Montréal & Mila  \nHsiu-Chin Lin [hsiu-chin.lin@mcgill. ca](hsiu-chin.lin@mcgill. ca)  \n[McGill University & Mila](McGill University & Mila)  \nAbstract  \nTraining reinforcement-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrained Markov decision process (MDP) formulations do. A standard mitigation hands control to a separate recovery policy whenever the agent leaves a designer-specified safe region (a subset of state space it should stay within), but the resulting mixed-policy rollouts silently bias every on-policy update, and the importance-sampling correction that would remove this bias is ill-defined whenever the recovery policy is deterministic. We address this bias with a drop-in modification of proximal policy optimization (PPO) . Its core is an unbiased policy-gradient estimator that uses the score function only at safe timesteps and never evaluates the recovery policy’s density, so it stays valid even when the recovery policy is deterministic, exactly where importance sampling breaks, and it empirically dominates importance sampling even when the recovery policy is stochastic. Because the recovery policy still makes credit assignment slow near the safe-region boundary, two further components accelerate learning: a closed-form value for recovery-triggering states when dynamicsand recovery are deterministic, and an imitation loss that copies recovery actions only when recovery succeeds. On a three-environment, five-seed benchmark, the resulting algorithm reduces training-time falls by factors of 233 × , 48×, and 26 × on HalfCheetah, Ant, and Unitree Go1 over standard PPO, while matching or exceeding PPO’s final reward, and on Ant, where the recovery policy is unreliable, it is the only method that reaches 80% of the best final reward.  \n1 Introduction  \nReinforcement learning (RL) policies often perform best when trained directly on the task they will be deployed on, but moving that training onto a physical robot is hard for a reason unrelated to asymptotic performance: learning requires failures, and on real hardware every failure has a price. A fall, a loss of balance that ends the episode, can damage the platform and cannot be undone the way a simulator reset can. Recent methods learn capable controllers by collecting hundreds of thousands of episodes in massively parallel simulation (Lee et al., 2020; Rudin et al., 2021; Agarwal et al. , 2022), where these failures are free, but on a real robot they are not (Ibarz et al., 2021; Smith et al., 2023b) . What gates real-world training is therefore not the final reward but the number of falls spent reaching it, and the goal of SafeExplorer is to keep that number small.  \nThe common way to keep learning on real hardware is to pair the agent with a separate recovery policy, a controller that takes over whenever the agent leaves a designer-specified safe region and steers the system back toward a nominal state. Because the safe region is set conservatively, the agent leaves it routinely, so the recovery policy fires far more often than a fall would occur, converting most would-be falls into cheap, reversible interventions. The controller is easy to assemble from off-the-shelf parts, an model predictive control (MPC) solver (Pua & Khadiv, 2024), a so","cbCairXNxLA4PgyE","https://ap.wps.com/l/cbCairXNxLA4PgyE","pdf",2269731,1,39,"English","en",105,"# Abstract\n# 1 Introduction\n## Problem: costly failures on physical robots\n## Recovery policies and the on-policy mismatch\n## Why importance sampling fails with deterministic recovery\n# Contributions","[{\"question\":\"How does SafeExplorer avoid the bias introduced by recovery interventions?\",\"answer\":\"SafeExplorer modifies PPO with an unbiased policy-gradient estimator that uses score-function information only at safe timesteps and never evaluates the recovery policy’s density. This keeps the gradient valid even when the recovery policy is deterministic, where importance sampling becomes ill-defined.\"}]",1784178178,98,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":27},"safeexplorer-an-unbiased-policy-gradient-for-reinforcement-learning-with-recovery-interventions","",{"@graph":35,"@context":77},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/safeexplorer-an-unbiased-policy-gradient-for-reinforcement-learning-with-recovery-interventions/82094/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"How does SafeExplorer avoid the bias introduced by recovery interventions?","Question",{"text":75,"@type":76},"SafeExplorer modifies PPO with an unbiased policy-gradient estimator that uses score-function information only at safe timesteps and never evaluates the recovery policy’s density. This keeps the gradient valid even when the recovery policy is deterministic, where importance sampling becomes ill-defined.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,112,115,120,123,127],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":45,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":45,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":45,"category_name":109,"show_sort_weight":110,"slug":111},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":113,"slug":114},30,"research-report",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":45,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":98,"slug":130},19,"General","general"]