Read and Reap the Rewards - Learning to Play Atari with the Help of Instruction Manuals

Oct 23, 2026

High sample complexity has long been a challenge for reinforcement learning (RL). Instruction manuals and wiki pages provide abundant unstructured text that can encode valuable features and policy guidance for specific tasks. This work proposes the Read and Reward framework, which accelerates Atari RL by reading developer-released manuals. It combines a QA Extraction module that extracts relevant manual information with a Reasoning module that evaluates object-agent interactions to provide auxiliary rewards to a standard A2C agent, improving performance and training speed across multiple RL algorithms.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Abstract

1 Introduction

Problem: sample inefficiency in RL

Hypothesis: instruction manuals as knowledge source

Read and Reward framework overview

arXiv :2302 .04449v4 [ cs .LG] 20 Jul 2024 Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals Yue Wu 1∗, Yewen Fan 1 , Paul Pu Liang 1 , Amos Azaria2 , Yuanzhi Li 1 ,3 , Tom Mitchell 1 1 Carnegie Mellon University, 2Ariel University, 3Microsoft Research ∗[ywu5@andrew.cmu.edu](ywu5@andrew.cmu.edu) Abstract High sample complexity has long been a challenge for RL. On the other hand, humans learn to perform tasks not only from interaction or demonstrations, but also by reading unstructured text documents, e.g., instruction manuals. Instruction manuals and wiki pages are among the most abundant data that could inform agents of valuable features and policies or task-specific environmental dynamicsand reward structures. Therefore, we hypothesize that the ability to utilize humanwritten instruction manuals to assist learning policies for specific tasks should lead to a more efficient and better-performing agent. We propose the Read and Reward framework. Read and Reward speeds up RL algorithms on Atari games by reading manuals released by the Atari game developers. Our framework consists of a QA Extraction module that extracts and summarizes relevant information from the manual and a Reasoning module that evaluates object-agent interactions based on information from the manual. An auxiliary reward is then provided toa standard A2C RL agent, when interaction is detected. Experimentally, various RL algorithms obtain significant improvement in performance and training speed when assisted by our design. Code [at github.com/Holmeswww/RnR](at github.com/Holmeswww/RnR). 1 Introduction Reinforcement Learning (RL) has achieved impressive performance in a variety of tasks, such as Atari (Mnih et al., 2015; Badia et al., 2020), Go (Silver et al., 2017), or autonomous driving (Fuchs et al., 2021; Kiran et al., 2022), and is hypothesized to be an important step toward artificial intelligence (Silver et al., 2021) . However, RL still faces a great challenge when applied to complicated, realworld-like scenarios (Shridhar et al., 2020b; Szot et al., 2021) due to its low sample efficiency. The Skiing game in Atari, for example, requires the skier (agent) to ski down a snowy hill and hit the objective gates in the shortest time, while avoiding obstacles. Such an intuitive game with simple controls (left, right) still required 80 billion frames to solve with existing RL algorithms (Badia et al., 2020), roughly equivalent to 100 years of non-stop playing. Observing the gap between RL and human performance, we identify an important cause: the lack of knowledge and understanding about the game. One of the most available sources of information, that is often used by humans to solve real-world problems, is unstructured text, e.g., books, instruction manuals, and wiki pages. In addition, unstructured language has been experimentally verified asa powerful and flexible source of knowledge for complex tasks (Tessler et al., 2021) . Therefore, we hypothesize that RL agents could benefit from reading and understanding publicly available instruction manuals and wiki pages. 37th Conference on Neural Information Processing Systems (NeurIPS 2023) . Figure 1: An overview of our Read and Reward framework. Our system receives the current frame in the environment, and the instruction manual as input. After object detection and grounding, the QA Extraction Module extracts and summarizes relevant information from the manual, and the Reasoning Module assigns auxiliary rewards to detected in-game events by reasoning with outputs from the QA Extraction Module. The “Yes/No” answers are then mapped to +5/ − 5 auxiliary rewards. In this work, we demonstrate the possibility for RL agents to benefit from human-written instruction manuals. We download per-game instruction manuals released by the original game developers 1 or Wikipedia and experiment with 4 Atari games where the object location ground-truths are available.

FAQs

What problem does the paper focus on in reinforcement learning?

It targets RL’s high sample complexity and low sample efficiency, especially in complicated, real-world-like scenarios.

How does the Read and Reward framework use instruction manuals?

It reads Atari instruction manuals and wiki-style text to extract relevant information and then reasons over object-agent interactions to assign auxiliary rewards.

What are the two main modules in the proposed framework?

A QA Extraction module summarizes relevant manual content, and a Reasoning module evaluates in-game interactions using outputs from the QA module.

Oct 23, 2026

Upload by MrHarris58