[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-142571-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-142571-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","learning-to-execute-actions-or-ask-clarification-questions-minecraft-corpus-dataset-extension-improved-collaborative-building-agent","Learning to Execute Actions or Ask Clarification Questions - Minecraft Corpus Dataset Extension - Improved Collaborative Building Agent","","Collaborative tasks rely on communication to achieve shared goals, and collaborative building in Minecraft requires both instruction following and proactive clarification. This paper extends the Minecraft Corpus Dataset by annotating builder utterances into eight dialogue types, including clarification questions, and introduces a builder agent model that decides when to execute versus ask. Experiments show state-of-the-art performance with substantial gains on collaborative building. Two additional learning tasks are defined: learning to ask and joint learning across both objectives.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/learning-to-execute-actions-or-ask-clarification-questions-minecraft-corpus-dataset-extension-improved-collaborative-building-agent/142571/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/learning-to-execute-actions-or-ask-clarification-questions-minecraft-corpus-dataset-extension-improved-collaborative-building-agent/142571.png","ImageObject",300,407,{"name":42,"@type":43},"Melati","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-18","2026-08-25",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",11,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does the paper address in Minecraft collaborative building?","Question",{"text":63,"@type":64},"It targets the limitation of existing Minecraft-based agents that focus on executing instructions but do not reliably ask clarification questions when information is ambiguous.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does the paper extend the Minecraft Corpus Dataset?",{"text":68,"@type":64},"It annotates all builder utterances into eight dialogue utterance types, explicitly including clarification questions, to support learning of proactive dialogue behavior.",{"name":70,"@type":61,"acceptedAnswer":71},"What new tasks does the paper define?",{"text":72,"@type":64},"It defines the learning to ask task and a joint learning task that solves both collaborative building and learning to ask together.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},142571,1787683594,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":55,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":80,"read_time":144,"preview_page_count":4},962085570644,"https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d","Learning to Execute Actions or Ask Clarification Questions  \nZhengxiang Shi and Yue Feng and Aldo Lipani University College London, London, UK {zhengxiang.shi.19,yue.feng.20, [aldo.lipani}@ucl.ac.uk](aldo.lipani}@ucl.ac.uk)  \narXiv :2204 .08373v2 [ cs .CL] 20 Apr 2022  \nAbstract  \nCollaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose anew builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also define two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly.  \n1 Introduction  \nFollowing instructions in natural language by intelligent agents to achieve a shared goal with the instructors in a pre-defined environment is a ubiquitous task in many scenarios, e.g., findinga target object in an environment (Nguyen and Daumé III, 2019 ; Roman et al., 2020), drawing a picture (Lachmy et al., 2021), or building a target structure (Narayan-Chen et al., 2019) . A number of machine learning (ML) research projects about following instructions tasks have been initiated by making use of the video game Minecraft (Johnson et al., 2016 ; Shu et al., 2018 ; Narayan-Chenet al., 2019 ; Guss et al., 2019 ; Jayannavar et al., 2020) . Building such agents requires to make  \nWorld State: Before the dialogue  \nDialogue Context  \nWorld State: After the dialogue  \nArchitect : Aparently, this first structure is a warm up Architect : It’s just 3 blocks arranged in an L shape  \nBuilder : Oh cool  \nArchitect : So two blocks next to each other on the ground  \nBuilder : Any color?  \nArchitect : And then one more block on top of either  \nArchitect : They are blue!  \nFigure 1: A simple example of builder task: The builder can observe the world state and dialogue context. For the sake of space, only a part of the dialogue history is displayed. The utterance in green displays understanding and the utterance in yellow asks a clarification question.  \nprogress in grounded natural language understanding – understanding complex instructions, for example, with spatial relations in natural language – self-improvement – studying how to flexibly learn from human interactions – synergies of ML components – exploring the integration of several MLand non-ML components to make them work together (Szlam et al., 2019) .  \nThe recently introduced Minecraft Corpus dataset (Narayan-Chen et al., 2019) proposes a collaborative building task, in which an architect anda builder can communicate via a textual chat. Architects are provided with a target structure they want to have built, and the builders are the only ones who can control the Minecraft avatar in the virtual environment. The task consists in building 3D structures in a block world-like scenario collaboratively, as shown in the Figure 1. Earlier works in Minecraft collaborative building tasks (Jayannavar et al., 2020) attempted to build an automated builder agent with a large action space but failed to allow the builder to take the initiative in the con-  \nversation. However, an intelligent agent should not only understand and execute the instructor","cbCaiaPcD4yJoi2E","https://ap.wps.com/l/cbCaiaPcD4yJoi2E","pdf",3690882,"English","# Introduction\n## Collaborative building in Minecraft\n## Need for proactive clarification\n## Extending the dataset with dialogue types\n# Proposed approach\n## Builder agent model and dialogue slots\n## Decision for asking vs executing\n# Experimental results\n## Learning to ask task\n## Joint learning task","[{\"question\":\"What problem does the paper address in Minecraft collaborative building?\",\"answer\":\"It targets the limitation of existing Minecraft-based agents that focus on executing instructions but do not reliably ask clarification questions when information is ambiguous.\"},{\"question\":\"How does the paper extend the Minecraft Corpus Dataset?\",\"answer\":\"It annotates all builder utterances into eight dialogue utterance types, explicitly including clarification questions, to support learning of proactive dialogue behavior.\"},{\"question\":\"What new tasks does the paper define?\",\"answer\":\"It defines the learning to ask task and a joint learning task that solves both collaborative building and learning to ask together.\"}]","Learning to Execute Actions or Ask Clarification Questions - Minecraft Corpus Dataset Extension - Improved Collaborative Building Agent | PDF",28]