Deep Reinforcement Learning with Active Vision on Atari Environments - Master’s thesis - A Study on Resolution and Field of View

Sep 18, 2026

Uploaded by Sarah

Reinforcement learning algorithms typically perform well in fully observable settings, and Atari 2600 benchmarks use the full screen as the input state. This thesis investigates whether restricting observations via active vision can improve efficiency by turning the state-space into a partially observable one. The agent is trained jointly to play the game and control its focus of attention using a modified recurrent A3C-LSTM network. Models simulating human visual system variants compare constant and decreasing resolution strategies and peripheral vision, showing varying gains across Pong, Breakout, and Beam Rider.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages58

Reading Time2 h 26 min

AccessFree to download

1 Introduction

2 Research Questions

3 Reinforcement Learning

3.1 Markov Process

3.2 Markov Reward Process

3.3 Markov Decision Process

3.4 TD-Learning

3.5 Q-Learning

3.6 Value Function Approximation

3.7 Policy Gradient

3.8 Actor-Critic

3.9 Advantage Function

3.10 Exploration vs. Exploitation

4 Deep Reinforcement Learning

4.1 The DQN Algorithm

4.2 The A3C Algorithm

4.3 A3C with LSTM

4.4 Other Extensions Included in A3C-LSTM

5 Focus-of-Attention

5.1 Modifications of Loss Function

5.2 Pre-processing and Modifications

5.3 Constant Resolution Vision Window

5.4 Decreasing Resolution Vision Window

5.5 Peripheral Vision

6 Experiments

6.1 Pong

6.2 Breakout

6.3 Beam Rider

6.4 Experiment Set-up

7 Results

8 Discussion

8.1 Non-FoA Models

8.2 Constant Resolution Models

8.3 Decreasing Resolution Models

8.4 Peripheral Vision

8.5 Analysis of Agent Behavior

9 Conclusions

A Appendix

A.1 Network Architecture

A.2 Pixel-information

A.3 Hyperparameters

MASTER’S THESIS 2022 Deep Reinforcement Learning with Active Vision on Atari Environments Robin Göransson ISSN 1650-2884 LU-CS-EX: 2022-46 DEPARTMENT OF COMPUTER SCIENC1ELTH | LUND UNIVERSITY EXAMENSARBETE Datavetenskap LU-CS-EX: 2022-46 Deep Reinforcement Learning with Active Vision on Atari Environments Djup Förstärkningsinlärning med Aktivt Seende på Atarimiljöer Robin Göransson Deep Reinforcement Learning with Active Vision on Atari Environments (A Study on Resolution and Field of View) Robin Göransson [ro0211go-s@student.lu.se](ro0211go-s@student.lu.se) June 20, 2022 Master’s thesis work carried out atthe Department of Computer Science, Lund University Supervisor: Volker Krüger, [volker.krueger@cs.lth.se](volker.krueger@cs.lth.se)[ ](volker.krueger@cs.lth.se)Examiner: Jacek Malec, [jacek.malec@cs.lth.se](jacek.malec@cs.lth.se) Abstract Reinforcement learning (RL) algorithms have throughout the years found most success on artificial domains where the state-space is fully observable. Atari 2600 games, where the full screen is used as the input state, are thus commonly used to evaluate the performance of an RL agent. Using the full screen as input could however be unnecessary as a significant amount of pixels on the screen do not contain any relevant information. In this thesis a current RL algorithm is expanded using active vision. By restricting the visible portion of the screen and giving the agent means to control its focus-ofattention the state-space is made partially observable. With the addition of active vision an agent must simultaneously learn to play the game and learn to control its focus-of-attention. This more challenging task is solved using a modified version of the recurrent A3C-LSTM network which can handle active vision. Throughout the thesis different models, that simulate the human visual system indifferent ways, were used. While all models used rectangular focal areas the models differed from each other in how resolution was handled. The first models used a single, constant resolution in the full focal area. By comparing these models it could be shown that a larger focal area using a lower resolution is preferable to a smaller focal area with higher resolution. Next, the resolution was set to decrease with increased distance to the center of the focal area. The model using the larger focal area and a lower resolution was replaced with a model using the same large focal area but with higher resolution towards the center and lower towards the edges. The decreasing resolution did not improve performance. As a final addition peripheral vision was added to the models. The peripheral vision was created using a very low resolution background outside the focal area. This addition did improve the performance of the models on Pong and Breakout while the addition barely affected the performance on Beam Rider. The model using decreasing resolution with the added peripheral was the best performing model on both Pong and Breakout while the most successful constant resolution model without peripheral achieved the best performance on Beam Rider. 2 Contents 1 Introduction 5 2 Research Questions 6 3 Reinforcement Learning 6 3.1 Markov Process ...................................... 6 3.2 Markov Reward Process ................................. 6 3.3 Markov Decision Process ................................. 7 3.4 TD-Learning ....................................... 8 3.5 Q-Learning ........................................ 8 3.6 Value Function Approximation ............................. 9 3.7 Policy Gradient ...................................... 9 3.8 Actor-Critic ........................................ 10 3.9 Advantage Function ................................... 10 3.10 Exploration vs. Exploitation .............................. 11 4 Deep Reinforcement Learning 12 4.1 The DQN Algorithm ................................... 12 4.2 The A3C Alg

FAQs

Why use active vision for Atari reinforcement learning instead of the full screen?

The thesis argues that many pixels in the full-screen state may not carry relevant information. Active vision restricts visible areas, making the state-space partially observable and potentially more efficient.

How is the agent trained under active vision?

The agent must simultaneously learn to play the game and learn where to focus its attention. This is achieved using a modified recurrent A3C-LSTM network that can handle active vision.

What visual variants were compared in the experiments?

The study compares models using constant resolution focal areas, decreasing resolution across the focal window, and adding peripheral vision via a low-resolution background outside the focal area.

Which model performed best across the Atari games studied?

The model combining decreasing resolution with added peripheral vision performed best on Pong and Breakout. For Beam Rider, the best result came from the most successful constant-resolution model without peripheral vision.

Sep 18, 2026

Uploaded by Sarah