[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-512111-105":59,"doc-detail-512111-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","decentralized-cooperative-reinforcement-learning-with-hierarchical-information-structure","Decentralized Cooperative Reinforcement Learning with Hierarchical Information Structure","","Multi-agent reinforcement learning faces information asymmetry that makes learning difficult when agents have different observations and effective environments change over time. This work studies a two-agent hierarchical information structure where a leader acts first and a follower acts after observing the leader, while both agents share the same reward (and state transition in MDPs) depending on joint actions. The paper develops decentralized hierarchical algorithms for two-agent multi-armed bandits and Markov decision processes, avoiding coordination and communication.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/decentralized-cooperative-reinforcement-learning-with-hierarchical-information-structure/512111/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/decentralized-cooperative-reinforcement-learning-with-hierarchical-information-structure/512111.png","ImageObject",300,407,{"name":92,"@type":93},"Sarah ","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-08","2026-10-06",true,{"@type":102,"interactionType":103,"userInteractionCount":29},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What information asymmetry problem does the paper address in multi-agent reinforcement learning?","Question",{"text":112,"@type":113},"Agents can have different observations, leading to a time-varying effective environment and non-convergence issues for independent learning.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How does the hierarchical information structure work in the two-agent model?",{"text":117,"@type":113},"The leader chooses an action first, then the follower selects an action after observing the leader’s action, based on joint outcomes.",{"name":119,"@type":110,"acceptedAnswer":120},"What is the key benefit of the proposed approach compared with coordination or communication methods?",{"text":121,"@type":113},"The algorithms require no coordination or communication, yet still achieve near-optimal regret bounds.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},512111,1791289946,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":29,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":139,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145},962085320529,"https://ap-avatar.wpscdn.com/davatar_9964176cb1d06d4a9deccf72a44ae3dc","Decentralized Cooperative Reinforcement Learning with Hierarchical Information Structure  \nHsu Kao HSUKAO @UMICH . EDU  \nUniversity of Michigan  \nChen-Yu Wei CHENYU . WEI @USC . EDU  \nUniversity of Southern California  \nVijay Subramanian VGSUBRAM @UMICH . EDU  \nUniversity of Michigan  \nEditors: Sanjoy Dasgupta and Nika Haghtalab  \nAbstract  \nMulti-agent reinforcement learning (MARL) problems are challenging due to information asymmetry. To overcome this challenge, existing methods often require high level of coordination or communication between the agents. We consider two-agent multi-armed bandits (MABs) and Markov decision processes (MDPs) with a hierarchical information structure arising in applications, which we exploit to propose simpler and more efficient algorithms that require no coordination or communication. In the structure, in each step the “leader” chooses her action first, and then the “follower”decides his action after observing the leader’s action. The two agents observe the same reward (and the same state transition in the MDP setting) that depends on their joint action. For the bandit setting, we propose  a hierarchical bandit algorithm that achieves a near-optimal gap-independent regtheWerofmbrthese(ABf acendoo)ands ofhecataearleofa-opdemilmantipaldlegathfollpenlowers deeran,ntrdreespthgeeilOyw,ita(lonhTd(Te)sp),irure Ambchye, dofheeese robte Hmateetarnhe-opumextibimerstialofngrbpboueroundepndsisi.oeethSmoMthfDeAPn, , nrdgoT,fwe ostatet,ainTih(H7 Snumb2eABTr of ep) iegodts, Keywords: hierarchical information structure, multi-agent online learning, multi-armed bandit, Markov decision process  \n1. Introduction  \nMulti-agent reinforcement learning (MARL) has received great attention due to its wide variety of applications and the tremendous advances in single-agent RL techniques (Zhang et al., 2020a) . In a multi-agent environment, each agent has different observations and may have different sets of information. This is referred to as the information asymmetry property (Chang et al., 2021) . One straightforward method used with information asymmetry is to let every agent concurrently learn based on its own information using single-agent algorithms. However, this creates the nonstationarity issue since the effective environment observed by each agent is time-varying, which sometimes causes non-convergence of the algorithms. Another line of solutions is to enforce coordination among agents, essentially transforming a multi-agent system back to (or making it more similar to) a single-agent one. One way to achieve this is through communication (Shahrampour et al., 2017 ; Zhang et al., 2021a), which introduces extra costs that may be intolerable in some  \n© 2022 H. Kao, C.-Y. Wei & V. Subramanian.  \nDECENTRALIZED COOPERATIVE RL WITH HIERARCHICAL INFORMATION STRUCTURE  \ncases. A more broadly applicable scheme is through the common information (CI) approach (Nayyar et al., 2013 ; Chang et al., 2021 ; Dibangoye and Buffet, 2018) . The CI approach relies on a set of CI shared by all agents, and all agents need to agree on a protocol that specifies the joint policy updates of all agents upon receiving a certain piece of CI. With this protocol, an agent may be able to infer the actions taken by other agents without observing them. However, this approach has several shortcomings, making it hard to apply in practice: it has high computational complexity (since every agent has to perform policy updates for all the other agents), and it requires all agents to have very tight coordination (e.g., sharing randomization seeds in each round and knowing all details in the algorithms of other agents), which may be infeasible if synchronization among agents or agent privacy is an issue.  \nIn this paper, we address the aforementioned issues in a special but widely applicable MARL setting. We consider MARL team problems 1 with a particular hierarchical information structure between agents under the settings of multi-agent multi-armed band","cbCaiaWE8iYILStZ","https://ap.wps.com/l/cbCaiaWE8iYILStZ","pdf",938231,33,"English","# Abstract\n# Introduction\n## Information asymmetry in MARL\n## Coordination via communication and common information (CI)\n## Hierarchical information structure in two-agent settings\n# Problem setting and algorithm outline","[{\"question\":\"What information asymmetry problem does the paper address in multi-agent reinforcement learning?\",\"answer\":\"Agents can have different observations, leading to a time-varying effective environment and non-convergence issues for independent learning.\"},{\"question\":\"How does the hierarchical information structure work in the two-agent model?\",\"answer\":\"The leader chooses an action first, then the follower selects an action after observing the leader’s action, based on joint outcomes.\"},{\"question\":\"What is the key benefit of the proposed approach compared with coordination or communication methods?\",\"answer\":\"The algorithms require no coordination or communication, yet still achieve near-optimal regret bounds.\"}]","Decentralized Cooperative Reinforcement Learning with Hierarchical Information Structure | PDF",1791265330,83]