[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-269473-105":59,"doc-detail-269473-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","language-models-are-unsupervised-multitask-learners-zero-shot-learning","Language Models are Unsupervised Multitask Learners - Zero-shot learning","","Natural language processing tasks such as question answering, machine translation, reading comprehension, and summarization are commonly solved with supervised learning on task-specific datasets. This work shows that language models learn these tasks without explicit supervision when trained on WebText, a dataset of millions of webpages. Conditioning on documents and questions enables strong answers on CoQA, matching or exceeding major baselines without 127,000+ training examples.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/language-models-are-unsupervised-multitask-learners-zero-shot-learning/269473/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/language-models-are-unsupervised-multitask-learners-zero-shot-learning/269473.png","ImageObject",300,407,{"name":92,"@type":93},"Gelato","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-03","2026-09-15",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",12,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"How do the language models learn tasks without explicit supervision?","Question",{"text":113,"@type":114},"They are trained on WebText, and during inference they are conditioned on the document plus questions to generate answers, showing task capability without separate supervised task datasets.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"What tasks and evaluation benchmark are highlighted in the study?",{"text":118,"@type":114},"The paper focuses on NLP tasks including question answering and evaluates question answering performance using the CoQA dataset.",{"name":120,"@type":111,"acceptedAnswer":121},"Why does increasing model capacity matter for zero-shot task transfer?",{"text":122,"@type":114},"Model capacity is identified as essential; scaling the largest model improves performance in a log-linear fashion across tasks in the zero-shot setting.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},269473,1789452261,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":130,"read_time":26,"preview_page_count":4},19241457091524,"https://us-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c","Language Models are Unsupervised Multitask Learners  \nAlec Radford * 1 Jeffrey Wu * 1 Rewon Child 1 David Luan 1 Dario Amodei ** 1 Ilya Sutskever ** 1  \nAbstract  \nNatural language processing tasks, such as question answering, machine translation, reading comprehension, and summarization, are typically approached with supervised learning on taskspecific datasets. We demonstrate that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText. When conditioned on a document plus questions, the answers generated by the language model reach 55 F1 on the CoQA dataset-matching or exceeding the performance of 3 out of 4 baseline systems without using the 127,000+ training examples.  \nThe capacity of the language model is essential to the success of zero-shot task transfer and increasing it improves performance in a log-linear fashion across tasks. Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain coherent paragraphs of text. These findings suggest a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.  \n1. Introduction  \nMachine learning systems now excel (in expectation) at tasks they are trained for by using a combination of large datasets, high-capacity models, and supervised learning (Krizhevsky et al., 2012) (Sutskever et al., 2014) (Amodeiet al., 2016) . Yet these systems are brittle and sensitive to slight changes in the data distribution (Recht et al., 2018) and task specification (Kirkpatrick et al., 2017) . Current systems are better characterized as narrow experts rather than  \n*, **Equal contribution 1 OpenAI, San Francisco, California, United States. Correspondence to: Alec Radford \u003C[alec@openai.com](alec@openai.com) >.  \ncompetent generalists. We would like to move towards more general systems which can perform many tasks – eventually without the need to manually create and label a training dataset for each one.  \nThe dominant approach to creating ML systems is to collect a dataset of training examples demonstrating correct behavior for a desired task, train a system to imitate these behaviors, and then test its performance on independent and identically distributed (IID) held-out examples. This has served well to make progress on narrow experts. But the often erratic behavior of captioning models (Lake et al., 2017), reading comprehension systems (Jia & Liang, 2017), and image classifiers (Alcorn et al., 2018) on the diversity and variety of possible inputs highlights some of the shortcomings of this approach.  \nOur suspicion is that the prevalence of single task training on single domain datasets is a major contributor to the lack of generalization observed in current systems. Progress towards robust systems with current architectures is likely to require training and measuring performance on a wide range of domains and tasks. Recently, several benchmarkshave been proposed such as GLUE (Wang et al., 2018) and decaNLP (McCann et al., 2018) to begin studying this.  \nMultitask learning (Caruana, 1997) is a promising framework for improving general performance. However, multitask training in NLP is still nascent. Recent work reports modest performance improvements (Yogatama et al., 2019) and the two most ambitious efforts to date have trained on a total of 10 and 17 (dataset, objective) pairs respectively (McCann et al., 2018) (Bowman et al., 2018) . From a meta-learning perspective, each (dataset, objective) pair is a single training example sampled from the distribution of datasets and objectives. Current ML systems need hundreds to thousands of examples to induce functions which generalize well. This suggests that multitask training many need just as man","cbCaipn8f9LvI08q","https://ap.wps.com/l/cbCaipn8f9LvI08q","pdf",582775,24,"English","# Abstract\n# Introduction\n## Motivation for multitask learning\n## Limitations of supervised task training\n## Multitask learning and meta-learning perspective\n## Prior work and transfer approaches","[{\"question\":\"How do the language models learn tasks without explicit supervision?\",\"answer\":\"They are trained on WebText, and during inference they are conditioned on the document plus questions to generate answers, showing task capability without separate supervised task datasets.\"},{\"question\":\"What tasks and evaluation benchmark are highlighted in the study?\",\"answer\":\"The paper focuses on NLP tasks including question answering and evaluates question answering performance using the CoQA dataset.\"},{\"question\":\"Why does increasing model capacity matter for zero-shot task transfer?\",\"answer\":\"Model capacity is identified as essential; scaling the largest model improves performance in a log-linear fashion across tasks in the zero-shot setting.\"}]","Language Models are Unsupervised Multitask Learners - Zero-shot learning | PDF"]