Learning to Execute Actions or Ask Clarification Questions - NAACL 2022 Findings

Aug 25, 2026

Uploaded by Tawan

Collaborative tasks rely on communication to achieve shared goals, and collaborative building in Minecraft requires agents to both follow requests and manage ambiguity. This paper develops an intelligent builder agent that can initiate dialogue by asking clarification questions when information is insufficient. It extends the Minecraft Corpus Dataset by annotating builder utterances into eight types, including clarification questions, and introduces a model to decide when to ask or execute. Results show state-of-the-art performance and define two new learning tasks, including joint learning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Introduction

Minecraft Corpus dataset and collaborative building

Motivation for clarification questions

Dataset annotation and dialogue utterance types

Proposed builder agent and learning tasks

Learning to Execute Actions or Ask Clarification Questions Zhengxiang Shi and Yue Feng and Aldo Lipani University College London, London, UK {zhengxiang.shi.19,yue.feng.20, [aldo.lipani}@ucl.ac.uk](aldo.lipani}@ucl.ac.uk) Abstract Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also define two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly. 1 Introduction Following instructions in natural language by intelligent agents to achieve a shared goal with the instructors in a pre-defined environment is a ubiquitous task in many scenarios, e.g., finding a target object in an environment (Nguyen and Daumé III, 2019 ; Roman et al., 2020), drawing a picture (Lachmy et al., 2021), or building a target structure (Narayan-Chen et al., 2019) . A number of machine learning (ML) research projects about following instructions tasks have been initiated by making use of the video game Minecraft (Johnson et al., 2016 ; Shu et al., 2018 ; Narayan-Chenet al., 2019 ; Guss et al., 2019 ; Jayannavar et al., 2020) . Building such agents requires to make World State: Before the dialogue Dialogue Context World State: After the dialogue Architect : Aparently, this ﬁrst structure is a warm up Architect : It’s just 3 blocks arranged in an L shape Builder : Oh cool Architect : So two blocks next to each other on the ground Builder : Any color? Architect : And then one more block on top of either Architect : They are blue! Figure 1: A simple example of builder task: The builder can observe the world state and dialogue context. For the sake of space, only a part of the dialogue history is displayed. The utterance in green displays understanding and the utterance in yellow asks a clarification question. progress in grounded natural language understanding – understanding complex instructions, for example, with spatial relations in natural language – self-improvement – studying how to flexibly learn from human interactions – synergies of ML components – exploring the integration of several MLand non-ML components to make them work together (Szlam et al., 2019) . The recently introduced Minecraft Corpus dataset (Narayan-Chen et al., 2019) proposes a collaborative building task, in which an architect anda builder can communicate via a textual chat. Architects are provided with a target structure they want to have built, and the builders are the only ones who can control the Minecraft avatar in the virtual environment. The task consists in building 3D structures in a block world-like scenario collaboratively, as shown in the Figure 1. Earlier works in Minecraft collaborative building tasks (Jayannavar et al., 2020) attempted to build an automated builder agent with a large action space but failed to allow the builder to take the initiative in the conversation. However, an intelligent agent should 2060 Findings of the Association for Computational Linguistics: NAACL 2022 , pag

FAQs

What problem does the paper address in collaborative building agents?

It addresses the need for agents to not only execute instructions but also ask clarification questions when the instructions are ambiguous.

How does the paper extend the Minecraft Corpus Dataset?

It annotates all builder utterances into eight dialogue utterance types, including clarification questions.

What is the main contribution of the proposed builder agent model?

The model learns to map instructions to actions and determines when to ask clarification questions versus execute instructions, improving performance in collaborative building.

Aug 25, 2026

Uploaded by Tawan