Improving Single and Multi-Agent Deep Reinforcement Learning Methods - Thesis abstract

Aug 4, 2026

Upload by Violet

Reinforcement Learning (RL) trains an agent to make decisions via rewards or penalties from environment interactions, and Deep RL combines RL with deep neural networks to handle complex, high-dimensional data for long-horizon sequential decisions. This thesis examines obstacles that hinder learning in specific environments and proposes methods to improve performance, sample efficiency, and generalizability. For single-agent RL, it addresses sparse-reward exploration using semantic exploration. For cooperative multi-agent RL, it tackles coordination and joint-action exploration with universal value exploration, scalable role-based learning, and analysis of independent policy methods.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages219

Reading Time9 h 12 min

AccessFree to download

Abstract

Single-agent exploration

Cooperative multi-agent reinforcement learning

Summary of contributions

Improving Single and Multi-Agent Deep Reinforcement Learning Methods Tarun Gupta Exeter College University of Oxford A thesis submitted for the degree of Doctor of Philosophy Michaelmas Term 2023 2 Abstract Reinforcement Learning (RL) is a framework where an agent learns to make decisions using data-driven feedback from interactions with the environment in the form of rewards or penalties for actions. Deep RL integrates deep learning with RL, harnessing the power of deep neural networks to process complex, high-dimensional data. Using the framework of deep RL, our machine learning research community has achieved tremendous progress in enabling machines to make sequential decisions over long time horizons. These advances include attaining super-human performance in Atari [Mnih et al., 2015], mastering the game of Go, beating the human worldchampion [Silver et al., 2017], providing robust recommendation systems [GomezUribe and Hunt, 2015, Singh et al., 2021] . This thesis focuses on identifying some key challenges that impede the learning of RL agents within their specific environmentsand improving the methods leading to better performance of agents, improved sample efficiency, and generalizability of learned agent policies. In Part I of the thesis, we focus on exploration in single-agent RL settings where an agent must interact with a complex environment to pursue a goal. An agent that fails to explore its environment is unlikely to achieve high performance, as it will miss critical rewards and, as a result, cannot learn optimal behavior. One key challenge arises from sparse reward environments where the agent only receives feedback once the task is completed, making exploration more challenging. We propose a novel method that enables semantic exploration, resulting in higher sample efficiency and performance on sparse reward tasks. In Part II of the thesis, we focus on cooperative Multi-Agent Reinforcement Learning (MARL), an extension of the usual RL setting, where we consider multiple agents interacting in the same environment toward a shared task. In multi-agent tasks requiring significant coordination among agents with strict penalties formiscoordination, state-of-the-art MARL methods often fail to learn useful behaviors as agents get stuck in a sub-optimal equilibrium. Another challenge is exploration in the joint action space of all agents, which grows exponentially with the number of agents. To address these challenges, we propose innovative approaches like universal value exploration and scalable role-based learning. These methods facilitate improved coordination among agents, faster exploration, and enhance the agents’ ability to adapt to new environments and tasks, showcasing zero-shot generalization capabilities and resulting in higher sample efficiency. Lastly, we investigate independent policybased methods in cooperative MARL, where each agent considers other agents as part of the environment. We show that such methods can perform better than state-of-the-art joint learning approaches on a popular multi-agent benchmark. In summary, the contributions of this thesis significantly improve the stateof-the-art in deep (multi-agent) reinforcement learning. The agents developed in this thesis can explore their environments efficiently to improve sample efficiency, learn tasks that require significant multi-agent coordination, and enable zero-shot generalization across various tasks. Improving Single and Multi-Agent Deep Reinforcement Learning Methods Tarun Gupta Exeter College University of Oxford A thesis submitted for the degree of Doctor of Philosophy Michaelmas Term 2023 Dedicated to my family Acknowledgements I am grateful to my supervisor, Shimon Whiteson for his unwavering support, guidance, and encouragement throughout my PhD journey. His insights and expertise have been invaluable, and I feel fortunate to have had the opportunity to work under his mentorship. I extend my hea

FAQs

What is the core focus of this thesis in deep reinforcement learning?

The thesis identifies challenges that limit RL agent learning in their environments and develops methods to improve agent performance, sample efficiency, and generalizability.

How does the thesis address exploration in single-agent RL?

It targets sparse-reward settings where feedback appears only after task completion, proposing a semantic exploration method to improve sample efficiency and performance.

What challenges arise in cooperative multi-agent RL, and how are they addressed?

The thesis highlights coordination difficulties and exploration in the exponentially growing joint action space. It proposes approaches such as universal value exploration and scalable role-based learning to enhance coordination, exploration, and adaptation, including zero-shot generalization.

Aug 4, 2026

Upload by Violet