[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85422-en":3,"doc-seo-85422-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85422,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","Adaptive Reinforcement Learning for Unobservable Random Delays","Standard reinforcement learning assumes instantaneous state observation and immediate action execution, an assumption that fails in cyber-physical and other real-world dynamic environments where interaction delays occur. Delays may be stochastic, time-varying, and unobservable at decision time. This work proposes an interaction layer framework that prepares a matrix of candidate future actions over a delay horizon. It enables adaptive handling of unpredictable delays and lost packets. An Actor-Critic method, ACDA, is developed and shows strong gains on locomotion benchmarks, including environments with measured delays.","Adaptive Reinforcement Learning for Unobservable Random Delays  \nJohn Wikman 1 Alexandre Proutiere 1 David Broman 1  \narXiv :2506 . 144 1 1v2 [ cs .LG] 12 Jul 2026  \nAbstract  \nIn standard reinforcement learning (RL) settings, the interaction between the agent and the environment is typically modeled as a Markov decision process (MDP), which assumes that the agent observes the system state instantaneously, selects an action without delay, and executes it immediately. In real-world dynamic environments, such as cyber-physical systems, this assumption often breaks down due to delays in the interaction between the agent and the system. These delays can vary stochastically overtime and are typically unobservable when deciding on an action. Existing methods deal with this uncertainty conservatively by assuming a known fixed upper bound on the delay, even if the delay is often much lower. In this work, we introduce the interaction layer, a general framework that enables agents to adaptively handle unobservable and time-varying delays. Specifically, the agent generates a matrix of possible future actions, anticipating a horizon of potential delays, to handle both unpredictable delays and lost action packets sent over networks. Building on this framework, we develop a modelbased algorithm, Actor-Critic with Delay Adaptation (ACDA), which dynamically adjusts to delay patterns. Our method significantly outperforms state-of-the-art approaches across a wide range of locomotion benchmark environments, including real-world measured delays.  \n1. Introduction  \nState-of-the-art reinforcement learning (RL) algorithms, such as Proximal Policy Optimization (PPO) (Schulmanet al., 2017) and Soft Actor-Critic (SAC) (Haarnoja et al.,  \n1EECS and Digital Futures, KTH Royal Institute of Technology, Stockholm, Sweden. Correspondence to: John Wikman \u003C[jwikman@kth.se](jwikman@kth.se) >.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \n2018), are typically built on the assumption that the environment can be modeled as a Markov decision process (MDP) . This framework implicitly assumes that the agent observes the current state instantaneously, selects an action without delay, and executes it immediately.  \nThis assumption often breaks down in real-world systems due to interaction delays that arise from various sources: the time taken to collect and transmit observations, the computation time needed for the agent to select an action, and the transmission and actuation delay when executing that action in the environment (as illustrated in Figure 1) . Delays pose no issue if the state of the environment is not evolving between its observation and the execution of the selected action. But in continuously evolving systems, such as robots operating in the physical world, the environment’s state may have changed by the time the action is executed (Brooks & Leondes, 1972) . Delays have been recognized as a key concern when applying RL to cyber-physical systems (Tan et al., 2018) . Outside the scope of RL, delays have also been studied in classic control (Ray, 1988 ; Luck & Ray, 1990) .  \n\n| Cloud Computer | Action | Embedded System | Excited System |\n| --- | --- | --- | --- |\n| \u003Cbr>\u003Cbr> |  | \u003Cbr>\u003Cbr>Act\u003Cbr>\u003Cbr>Read | ate\u003Cbr>ensors\u003Cbr> |\n|  | τapply τobserve |  |  |\n|  | Observation |  |  |\n\nu  \nS  \nFigure 1. Illustration of a setup affected by interaction delays. Any delay between the embedded system and the excited system is considered negligible or otherwise accounted for (further discussed in Appendix G.1) . The factors contributing to interaction delay are τobserve (τo ), τcompute (τc ), and τapply (τa ) .  \nThese interaction delays can be implicitly modeled by altering the transition dynamics of the MDP to form a partially observable Markov decision process (POMDP), in which the agent only receives outdated sensor observations. While this approach is practical and","cbCaihQgOxJVJejy","https://ap.wps.com/l/cbCaihQgOxJVJejy","pdf",4908469,1,66,"English","en",105,"# Abstract\n# Introduction\n## Background: MDP assumptions and delay challenges\n## Conservative fixed-delay approaches\n## Unobservable, time-varying delays and limitations\n## Contributions and proposed interaction layer\n## Actor-Critic with Delay Adaptation (ACDA)","[{\"question\":\"What problem does the paper address in reinforcement learning?\",\"answer\":\"The paper addresses the mismatch between standard RL assumptions and real systems where interaction delays occur between observation, action selection, and execution. These delays can be stochastic, time-varying, and unobservable at decision time.\"},{\"question\":\"Why is modeling delays with a POMDP not fully sufficient here?\",\"answer\":\"Treating delays as partially observable dynamics lets the agent use outdated sensor observations, but it limits the agent’s information about how the environment evolves during the delay period. This reduces access to critical evolution information.\"},{\"question\":\"How does the interaction layer framework work?\",\"answer\":\"The agent generates a matrix of possible future actions ahead of time, organized by a horizon of potential delay arrival times. This anticipates unpredictable delays and also accounts for lost action packets on networks.\"}]",1784203335,166,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"adaptive-reinforcement-learning-for-unobservable-random-delays","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/adaptive-reinforcement-learning-for-unobservable-random-delays/85422/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address in reinforcement learning?","Question",{"text":75,"@type":76},"The paper addresses the mismatch between standard RL assumptions and real systems where interaction delays occur between observation, action selection, and execution. These delays can be stochastic, time-varying, and unobservable at decision time.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why is modeling delays with a POMDP not fully sufficient here?",{"text":80,"@type":76},"Treating delays as partially observable dynamics lets the agent use outdated sensor observations, but it limits the agent’s information about how the environment evolves during the delay period. This reduces access to critical evolution information.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the interaction layer framework work?",{"text":84,"@type":76},"The agent generates a matrix of possible future actions ahead of time, organized by a horizon of potential delay arrival times. This anticipates unpredictable delays and also accounts for lost action packets on networks.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]