[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85420-en":3,"doc-seo-85420-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85420,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning","Model-free distributionally robust methods for infinite-horizon average-reward Markov decision processes are studied. The work develops non-asymptotic convergence analyses for robust Q-learning and actor–critic algorithms under contamination, total-variation, and Wasserstein uncertainty sets. Central to the analysis is proving that the robust Bellman operator is a strict contraction in a designed semi-norm, enabling stochastic approximation to learn the optimal robust Q-function with O(ε−2) dependence on target accuracy. Uniform robust TD convergence bounds support data-driven robust critic estimation and an ε-optimal robust policy with O(ε−2). Numerical simulations illustrate qualitative behavior, and results provide theoretical foundations for model-misspecification-robust planning and robust long-run policy learning from simulation.","Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learning  \nYang Xu 1 Swetha Ganesh 1 Vaneet Aggarwal 1  \n1 Purdue University, West Lafayette, Indiana, USA 47907  \narXiv :2506 .07040v4 [ cs .LG] 13 Jul 2026  \nAbstract  \nWe study model-free methods for distributionally robust infinite-horizon average-reward Markov decision processes (MDPs) . We present non-asymptotic convergence analyses of Qlearning and actor–critic algorithms for robust average-reward MDPs under contamination, total-variation distance, and Wasserstein uncertainty sets. A key ingredient of our analysis is showing that the optimal robust Bellman operator is a strict contraction with respect to a carefully designed semi-norm. This property enables a stochastic approximation update t˜hat learns the optimal robust Q-function with  \nO (ϵ−2) dependence on the target accuracy. We also establish robust TD convergence bounds whose constants are uniform over all stationary policies, yielding an efficient data-driven routine for robust critic estimation. Building on this, we introduce an actor–critic algorithm t˜hat learns an ϵ-optimal robust policy with  \nO (ϵ−2) dependence on the target accuracy.  \nWe provide numerical simulations to illustrate the qualitative behavior of the proposed algorithms. Our results contribute to the theoretical foundations of robust planning under model misspecification, and to model-free approaches for building robust long-run policies directly from simulation data.  \n1 INTRODUCTION  \nReinforcement learning (RL) has produced impressive results in fields such as robotics, finance, and healthcare by allowing agents to discover effective actions through interaction with their environments. Yet in many practical settings direct interaction is unsafe, pro-  \nhibitively costly, or constrained by strict data budgets Sünderhauf et al. [2018], Höfer et al. [2021] . Practitioners therefore train agents within simulated environments before deploying them into the physical world. This approach is a common choice for robotic control and autonomous driving. The unavoidable gap between a simulator and reality can nonetheless erode performance once the policy encounters dynamics that were absent during training. Robust RL mitigates this risk by framing learning as an optimization problem over a family of transition models, aiming for reliable behaviour under the most adverse member of that family. In this work, we focus on the setting where a planner interacts only with a fixed nominal simulator but seeksa policy that is robust to transition uncertainty after deployment.  \nReinforcement learning with an infinite horizon is usually studied under two reward formulations: the discounted formulation, which geometrically discounts future rewards, and the average-reward formulation, which maximizes the long-run mean return. Although the discounted objective has been widely used, its bias toward immediate gains can produce shortsighted policies in tasks that demand sustained efficiency. The average-reward formulation naturally fits these domains because each decision shapes cumulative performance over time. Typical examples include fleet management in ride-hailing, production scheduling in factories, and network resource allocation, where planners must optimize long-run throughput or service quality under uncertain dynamics. Yet the literature on robust reinforcement learning remains largely unexplored under the average-reward criterion.  \nExisting works on the robust average-reward RL is still sparse. Existing studies on analyzing Q-learning updates provide only asymptotic convergence results [Wang et al., 2023c, 2024b] . [Sun et al., 2024] examines the iteration complexity of vanilla policy gradient for the same problem but assumes an oracle that yields the  \nexact robust Q functions. Sample complexity studies on model-based line of work converts the average-reward objective into an equivalent discounted task and then applies","cbCaijKkSeSwY7RJ","https://ap.wps.com/l/cbCaijKkSeSwY7RJ","pdf",2257791,1,40,"English","en",105,"# Abstract\n# Introduction\n## Challenges and Contributions","[{\"question\":\"What is the problem setting of the document?\",\"answer\":\"The document studies model-free algorithms for infinite-horizon average-reward MDPs under distributional robustness, where the learner uses samples from a nominal simulator but aims for robustness to transition uncertainty after deployment.\"},{\"question\":\"Which uncertainty sets are considered for robust learning?\",\"answer\":\"It analyzes three common families of uncertainty sets: contamination sets, total-variation (TV) distance uncertainty sets, and Wasserstein distance uncertainty sets.\"},{\"question\":\"How does the paper obtain non-asymptotic convergence guarantees?\",\"answer\":\"It proves that the optimal robust Bellman operator is a strict contraction with respect to a carefully designed semi-norm, which enables stochastic approximation updates to learn the optimal robust Q-function and supports robust TD convergence bounds and actor–critic policy learning.\"}]",1784203287,101,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"efficient-q-learning-and-actor-critic-methods-for-robust-average-reward-reinforcement-learning","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/efficient-q-learning-and-actor-critic-methods-for-robust-average-reward-reinforcement-learning/85420/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the problem setting of the document?","Question",{"text":75,"@type":76},"The document studies model-free algorithms for infinite-horizon average-reward MDPs under distributional robustness, where the learner uses samples from a nominal simulator but aims for robustness to transition uncertainty after deployment.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Which uncertainty sets are considered for robust learning?",{"text":80,"@type":76},"It analyzes three common families of uncertainty sets: contamination sets, total-variation (TV) distance uncertainty sets, and Wasserstein distance uncertainty sets.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the paper obtain non-asymptotic convergence guarantees?",{"text":84,"@type":76},"It proves that the optimal robust Bellman operator is a strict contraction with respect to a carefully designed semi-norm, which enables stochastic approximation updates to learn the optimal robust Q-function and supports robust TD convergence bounds and actor–critic policy learning.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":21,"slug":118},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]