Learning to Execute Actions or Ask Clarification Questions - Minecraft Corpus Dataset Extension - Improved Collaborative Building Agent

Aug 25, 2026

Uploaded by Melati

Collaborative tasks rely on communication to achieve shared goals, and collaborative building in Minecraft requires both instruction following and proactive clarification. This paper extends the Minecraft Corpus Dataset by annotating builder utterances into eight dialogue types, including clarification questions, and introduces a builder agent model that decides when to execute versus ask. Experiments show state-of-the-art performance with substantial gains on collaborative building. Two additional learning tasks are defined: learning to ask and joint learning across both objectives.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

AccessFree to download

Introduction

Collaborative building in Minecraft

Need for proactive clarification

Extending the dataset with dialogue types

Proposed approach

Builder agent model and dialogue slots

Decision for asking vs executing

Experimental results

Learning to ask task

Joint learning task

Learning to Execute Actions or Ask Clariﬁcation Questions Zhengxiang Shi and Yue Feng and Aldo Lipani University College London, London, UK {zhengxiang.shi.19,yue.feng.20, [aldo.lipani}@ucl.ac.uk](aldo.lipani}@ucl.ac.uk) arXiv :2204 .08373v2 [ cs .CL] 20 Apr 2022 Abstract Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clariﬁcation questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clariﬁcations. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clariﬁcation questions, and propose anew builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also deﬁne two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly. 1 Introduction Following instructions in natural language by intelligent agents to achieve a shared goal with the instructors in a pre-deﬁned environment is a ubiquitous task in many scenarios, e.g., ﬁndinga target object in an environment (Nguyen and Daumé III, 2019 ; Roman et al., 2020), drawing a picture (Lachmy et al., 2021), or building a target structure (Narayan-Chen et al., 2019) . A number of machine learning (ML) research projects about following instructions tasks have been initiated by making use of the video game Minecraft (Johnson et al., 2016 ; Shu et al., 2018 ; Narayan-Chenet al., 2019 ; Guss et al., 2019 ; Jayannavar et al., 2020) . Building such agents requires to make World State: Before the dialogue Dialogue Context World State: After the dialogue Architect : Aparently, this ﬁrst structure is a warm up Architect : It’s just 3 blocks arranged in an L shape Builder : Oh cool Architect : So two blocks next to each other on the ground Builder : Any color? Architect : And then one more block on top of either Architect : They are blue! Figure 1: A simple example of builder task: The builder can observe the world state and dialogue context. For the sake of space, only a part of the dialogue history is displayed. The utterance in green displays understanding and the utterance in yellow asks a clariﬁcation question. progress in grounded natural language understanding – understanding complex instructions, for example, with spatial relations in natural language – self-improvement – studying how to ﬂexibly learn from human interactions – synergies of ML components – exploring the integration of several MLand non-ML components to make them work together (Szlam et al., 2019) . The recently introduced Minecraft Corpus dataset (Narayan-Chen et al., 2019) proposes a collaborative building task, in which an architect anda builder can communicate via a textual chat. Architects are provided with a target structure they want to have built, and the builders are the only ones who can control the Minecraft avatar in the virtual environment. The task consists in building 3D structures in a block world-like scenario collaboratively, as shown in the Figure 1. Earlier works in Minecraft collaborative building tasks (Jayannavar et al., 2020) attempted to build an automated builder agent with a large action space but failed to allow the builder to take the initiative in the con- versation. However, an intelligent agent should not only understand and execute the instructor

FAQs

What problem does the paper address in Minecraft collaborative building?

It targets the limitation of existing Minecraft-based agents that focus on executing instructions but do not reliably ask clarification questions when information is ambiguous.

How does the paper extend the Minecraft Corpus Dataset?

It annotates all builder utterances into eight dialogue utterance types, explicitly including clarification questions, to support learning of proactive dialogue behavior.

What new tasks does the paper define?

It defines the learning to ask task and a joint learning task that solves both collaborative building and learning to ask together.

Aug 25, 2026

Uploaded by Melati