[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83894-en":3,"doc-seo-83894-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83894,8796095461564,"Liam","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization","Reinforcement learning in real-world environments often degrades severely when feedback is delayed, since delayed observations may not reflect the true current state. Prior solutions reduce this issue by state augmentation or predicting true states, but they typically neglect the stochastic-MDP discrepancy between delayed and true states and its impact on optimal policies. This work proves the discrepancy exists, then introduces Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO) to estimate and use it for robust delayed policy weighting. Experiments on stochastic continuous robotic control confirm consistent gains, even with long and random delays.","Diffusion-Guided Uncertainty-Aware Delayed Policy Optimization  \nJunqi Tu 1 , Zejiao Liu2 , Fangfei Li2∗, Yang Tang 1 ∗  \n1The Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai, China  \n2The School of Mathematics, East China University of Science and Technology, Shanghai, China {23012389,[liuzejiao}@mail.ecust.edu.cn](liuzejiao}@mail.ecust.edu.cn), {lifangfei,[yangtang}@ecust.edu.cn](yangtang}@ecust.edu.cn)  \narXiv :2607 .05064v 1 [ cs .AI] 6 Jul 2026  \nAbstract  \nReinforcement learning in real world environments often suffers from severe performance degradation due to delayed feedback. Existing approaches typically mitigate performance degradation caused by observation delays by constructing augmented states or predicting the true states. However, these methods often overlook the inherent discrepancy between delayed state and true states induced by stochastic MDP. We theoretically prove the existence of such a discrepancy and show that it leads to the degradation of the optimal policy. To address this challenge, we propose Diffusion Guided Uncertainty Aware Delayed Policy Optimization (DUPO) . Our method explicitly models the relationship between delayed state message and the current state using a diffusion model, and leverages the resulting discrepancy estimates to weight delayed policies. Extensive experiments on continuous robotic control tasks with multiple stochastic delays demonstrate that DUPO consistently outperforms existing methods and remains effective even under long and random delay scenarios.  \nIntroduction  \nReinforcement learning (RL) has recently achieved remarkable progress in a variety of complex decision-making domains, including Go (Silver et al. 2018), competitive multiplayer online games (Berner et al. 2019), continuous control and simulated robotics (Haarnoja et al. 2018), as well as autonomous driving related tasks (Feng et al. 2023) . Despite these successes, an implicit assumption is often made that agent–environment interactions are instantaneous, such that actions take effect immediately and observations faithfully reflect the current state. In real-world systems, however, this assumption frequently breaks down due to communication latency, sensor pipelines, and the computational overhead of model inference. Without properly modeling and compensating for delays, RL agents may suffer severe degradation and even safety-critical failures (Paduraru et al. 2021); for instance, substantial performance loss in trading (Hasbrouck and Saar 2013), instability in dynamical systems (Dugard and Verriest 1998; Gu and Niculescu 2003), and reduced training efficiency and reproducibility in real-world robotics (Mahmood et al. 2018) . Therefore, explicitly incorporating action and observation delays into RL algorithm design is a crucial step towards reliable real-world deployment. Existing studies  \n∗Corresponding author.  \non delayed RL can be broadly categorized into three mainlines. Memoryless methods learn policies that depend only on the most recent observation, thereby ignoring the nonMarkovian nature induced by observation delays (Schuitema et al. 2010) . While simple to implement, these methods typically incur significant performance drops under large delays. Augmentation-based methods restore the Markov property by constructing an information state that concatenates the last observed state with the sequence of actions executed thereafter, reducing delayed RL to an augmented-state delayed MDP (Altman and Nain 1992; Katsikopoulos and Engelbrecht 2003; Bertsekas 2012), on which classical RL algorithms can be applied (Nath, Baranwal, and Khadilkar 2021) . However, as the delay horizon increases, the augmented state space expands rapidly, resulting in substantially higher sample complexity and pronounced instability in TD-style learning under long delays, reflecting a classic manifestation of the curse of dimensiona","cbCaisVMkr4CnwRN","https://ap.wps.com/l/cbCaisVMkr4CnwRN","pdf",9477618,1,14,"English","en",105,"# Abstract\n# Introduction\n# Existing Studies\n# Proposed Method (DUPO)","[{\"question\":\"Why do delayed observations cause performance degradation in reinforcement learning?\",\"answer\":\"Delayed feedback breaks the assumption that observations match the current state. The delay creates a discrepancy between delayed state information and the true current state, which can degrade the optimal policy.\"},{\"question\":\"What is the main contribution of DUPO?\",\"answer\":\"DUPO models the relationship between delayed state messages and the current true state using a diffusion model, producing discrepancy estimates that weight delayed policies during optimization.\"},{\"question\":\"How does DUPO handle stochastic or random delays?\",\"answer\":\"It explicitly accounts for delay-induced state discrepancy that grows with delay length in random-delay settings. The method avoids degradation from single-point state prediction and remains effective under long and random delays in experiments.\"}]",1784191275,35,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"diffusion-guided-uncertainty-aware-delayed-policy-optimization","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/diffusion-guided-uncertainty-aware-delayed-policy-optimization/83894/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do delayed observations cause performance degradation in reinforcement learning?","Question",{"text":75,"@type":76},"Delayed feedback breaks the assumption that observations match the current state. The delay creates a discrepancy between delayed state information and the true current state, which can degrade the optimal policy.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is the main contribution of DUPO?",{"text":80,"@type":76},"DUPO models the relationship between delayed state messages and the current true state using a diffusion model, producing discrepancy estimates that weight delayed policies during optimization.",{"name":82,"@type":73,"acceptedAnswer":83},"How does DUPO handle stochastic or random delays?",{"text":84,"@type":76},"It explicitly accounts for delay-induced state discrepancy that grows with delay length in random-delay settings. The method avoids degradation from single-point state prediction and remains effective under long and random delays in experiments.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]