[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-135761-105":59,"doc-detail-135761-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","a-generalist-agent-gato-model-and-data-capabilities","A Generalist Agent - Gato - model and data capabilities","","Inspired by large-scale language modeling, the report presents Gato, a single generalist agent designed beyond text generation. The system operates as a multi-modal, multi-task, multi-embodiment policy using one transformer sequence model with shared weights. Gato can handle diverse environments such as Atari play, image captioning, dialogue, and robotic manipulation, selecting appropriate outputs like text, joint torques, or button presses. The paper details the model, tokenization, training setup, dataset serialization, and documents current capabilities across 604 tasks.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/a-generalist-agent-gato-model-and-data-capabilities/135761/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/a-generalist-agent-gato-model-and-data-capabilities/135761.png","ImageObject",300,407,{"name":92,"@type":93},"Logic","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-17","2026-08-21",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",16,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What is Gato and what problem does it address?","Question",{"text":113,"@type":114},"Gato is trained as a multi-modal sequence model that can output different action types. It decides what to emit—text, joint torques, or discrete controls—based on the context and task setting.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"How does Gato represent different data types for a single model?",{"text":118,"@type":114},"The report states that Gato serializes all modalities into a flat sequence of tokens. This allows the same transformer network to process text, images, proprioception, and both discrete and continuous actions.",{"name":120,"@type":111,"acceptedAnswer":121},"What hardware scale and training approach are used for the real-world robot setting?",{"text":122,"@type":114},"Gato is focused on an operating point around 1.2B parameters to enable real-time control of real-world robots. Training is described as purely supervised offline for the current iteration, with notes that RL training is also possible in principle.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},135761,1787319370,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":145},1099513958762,"https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1784791008015729253","arXiv :2205 .06175v3 [ cs .AI] 11 Nov 2022  \nA Generalist Agent  \nScott Reed*,†, Konrad Żołna* , Emilio Parisotto* , Sergio Gómez Colmenarejo†, Alexander Novikov, Gabriel Barth-Maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar and Nando de Freitas†  \n* Equal contributions,†Equal senior contributions, All authors are affiliated with DeepMind [reedscot@deepmind. com](reedscot@deepmind. com)[ ](reedscot@deepmind. com)Reviewed on OpenReview: [https://openreview.net/forum?id=1ikK0kHjvj](https://openreview.net/forum?id=1ikK0kHjvj)  \nAbstract  \nInspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy.  \nThe same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.  \nFigure 1: A generalist agent. Gato can sense and act with different embodiments across a wide range of environments using a single neural network with the same set of weights. Gato was trained on 604 distinct tasks with varying modalities, observations and action specifications.  \nAtari images and discrete actions  \nBatched and masked shifted targets  \nquestions  \nProprioception  \n Image  \n Text  \n Continuous actions  \n Discrete actions  \nFigure 2: Training phase of Gato. Data from different tasks and modalities is serialized into a flat sequence of tokens, batched, and processed by a transformer neural network akin to a large language model. Masking is used such that the loss function is applied only to target outputs, i.e. text and various actions.  \n1 Introduction  \nThere are significant benefits to using a single neural sequence model across all tasks. It reduces the need for hand crafting policy models with appropriate inductive biases for each domain. It increases the amount and diversity of training data since the sequence model can ingest any data that can be serialized into a flat sequence. Furthermore, its performance continues to improve even at the frontier of data, compute and model scale (Kaplan et al., 2020; Hoffmann et al., 2022) . Historically, generic models that are better at leveraging computation have also tended to overtake more specialized domain-specific approaches (Sutton, 2019), eventually.  \nIn this paper, we describe the current iteration of a general-purpose agent which we call Gato, instantiated as a single, large, transformer sequence model. With a single set of weights, Gato can engage in dialogue, caption images, stack blocks with a real robot arm, outperform humans at playing Atari games, navigate in simulated 3D environments, follow instructions, and more.  \nWhile no agent can be expected to excel in all imaginable control tasks, especially those far outside of its training distribution, we here test the hypothesis that training an agent which is generally capable on a large number of tasks is possible; and that this general agent can be adapted with little extra data to succeed at an even larger number of tasks. We hypothesize that such an agent can be obtained through scaling data, compute and model parameters, continually broadening the training distribution while maintaining performance, towards covering any task, behavior and embodiment of interest. In this setting, natural language can act as a common grounding across otherwise incompatible embodiments, unlocking combinatorial generalization to new behaviors.  \nWe focus our training at the operating point of model scale that allows real-time control of real-world robots, currently a","cbCaieRwgrauIBvh","https://ap.wps.com/l/cbCaieRwgrauIBvh","pdf",7662379,42,"English","# Introduction\n## Model\n### Tokenization\n## Architecture, loss, and deployment","[{\"question\":\"What is Gato and what problem does it address?\",\"answer\":\"Gato is trained as a multi-modal sequence model that can output different action types. It decides what to emit—text, joint torques, or discrete controls—based on the context and task setting.\"},{\"question\":\"How does Gato represent different data types for a single model?\",\"answer\":\"The report states that Gato serializes all modalities into a flat sequence of tokens. This allows the same transformer network to process text, images, proprioception, and both discrete and continuous actions.\"},{\"question\":\"What hardware scale and training approach are used for the real-world robot setting?\",\"answer\":\"Gato is focused on an operating point around 1.2B parameters to enable real-time control of real-world robots. Training is described as purely supervised offline for the current iteration, with notes that RL training is also possible in principle.\"}]","A Generalist Agent - Gato - model and data capabilities | PDF",106]