[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-283272-105":3,"detail-sidebar-cat-1-en-105":81,"doc-detail-283272-en":126},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","llama-open-and-efficient-foundation-language-models-abstract","LLaMA: Open and Efficient Foundation Language Models - Abstract","","LLaMA introduces a family of foundation language models spanning 7B to 65B parameters. The training uses trillions of tokens and demonstrates that competitive state-of-the-art performance can be achieved using exclusively publicly available datasets, avoiding proprietary inaccessible data. LLaMA-13B surpasses GPT-3 (175B) on most benchmarks, while LLaMA-65B remains competitive with leading models such as Chinchilla-70B and PaLM-540B, and all models are released for research.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/template/","Template",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/template/presentations/","Presentations",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/template/llama-open-and-efficient-foundation-language-models-abstract/283272/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/llama-open-and-efficient-foundation-language-models-abstract/283272.png","ImageObject",442,249,{"name":42,"@type":43},"Bill Black","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-04","2026-09-16",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",16,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What is LLaMA, and what model sizes does it include?","Question",{"text":63,"@type":64},"LLaMA is a collection of foundation language models ranging from 7B to 65B parameters.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does LLaMA differ from many existing large language models in training data?",{"text":68,"@type":64},"LLaMA uses only publicly available data, aiming to be compatible with open-sourcing, unlike models that rely on non-public or undocumented datasets.",{"name":70,"@type":61,"acceptedAnswer":71},"What performance comparisons does the abstract highlight?",{"text":72,"@type":64},"LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},283272,1791729335,{"code":4,"msg":82,"data":83},"success",[84,88,93,98,103,107,112,117,122],{"id":85,"doc_module":22,"doc_module_name":25,"category_name":29,"show_sort_weight":86,"slug":87},11,90,"presentations",{"id":89,"doc_module":22,"doc_module_name":25,"category_name":90,"show_sort_weight":91,"slug":92},12,"Resumes",80,"resumes",{"id":94,"doc_module":22,"doc_module_name":25,"category_name":95,"show_sort_weight":96,"slug":97},14,"Invoices",70,"invoices",{"id":99,"doc_module":22,"doc_module_name":25,"category_name":100,"show_sort_weight":101,"slug":102},15,"Posters",60,"posters",{"id":55,"doc_module":22,"doc_module_name":25,"category_name":104,"show_sort_weight":105,"slug":106},"Social Media",50,"social-media",{"id":108,"doc_module":22,"doc_module_name":25,"category_name":109,"show_sort_weight":110,"slug":111},17,"Forms",40,"forms",{"id":113,"doc_module":22,"doc_module_name":25,"category_name":114,"show_sort_weight":115,"slug":116},18,"Letters",30,"letters",{"id":118,"doc_module":22,"doc_module_name":25,"category_name":119,"show_sort_weight":120,"slug":121},21,"Paper Templates",5,"papers-templates",{"id":123,"doc_module":22,"doc_module_name":25,"category_name":124,"show_sort_weight":4,"slug":125},158,"General","general-158",{"code":4,"msg":82,"data":127},{"doc_id":79,"user_id":128,"nickname":42,"user_avatar":129,"doc_module":22,"category_id":85,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":135,"language":136,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":137,"faqs":138,"seo_title":139,"seo_description":12,"update_tm":140,"read_time":141},24189269381491,"https://ap-avatar.wpscdn.com/avatar/160000cf11732dd8392?x-image-process=image/resize,m_fixed,w_180,h_180&k=1788146458752108895","LLaMA: Open and Efficient Foundation Language Models  \nHugo Touvron􀀃, Thibaut Lavril􀀃, Gautier Izacard􀀃, Xavier Martinet  \nMarie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin  \nEdouard Grave􀀃, Guillaume Lample􀀃  \nMeta AI  \narXiv :2302 . 13971v1 [ cs .CL] 27 Feb 2023  \nAbstract  \nWe introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA- 65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community 1.  \n1 Introduction  \nLarge Languages Models (LLMs) trained on massive corpora of texts have shown their ability to perform new tasks from textual instructions or from a few examples (Brown et al., 2020) . These few-shot properties first appeared when scaling models to asufficient size (Kaplan et al., 2020), resulting in aline of work that focuses on further scaling these models (Chowdhery et al., 2022 ; Rae et al., 2021) . These efforts are based on the assumption that more parameters will lead to better performance. However, recent work from Hoffmann et al. (2022) shows that, for a given compute budget, the best performances are not achieved by the largest models, but by smaller models trained on more data.  \nThe objective of the scaling laws from Hoffmann et al. (2022) is to determine how to best scale the dataset and model sizes for a particular training compute budget. However, this objective disregards the inference budget, which becomes critical when serving a language model at scale. In this context, given a target level of performance, the preferred model is not the fastest to train but the fastest at inference, and although it may be cheaper to train a large model to reach a certain level of  \n􀀃 Equal contribution. Correspondence: {htouvron, thibautlav,gizacard,egrave,[glample}@meta.com](glample}@meta.com)  \n1 [https://github.com/facebookresearch/llama](https://github.com/facebookresearch/llama)  \nperformance, a smaller one trained longer will ultimately be cheaper at inference. For instance, although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens.  \nThe focus of this work is to train a series of language models that achieve the best possible performance at various inference budgets, by training on more tokens than what is typically used. The resulting models, called LLaMA, ranges from 7B to 65B parameters with competitive performance compared to the best existing LLMs. For instance, LLaMA-13B outperforms GPT-3 on most benchmarks, despite being 10􀀂 smaller. We believe that this model will help democratize the access and study of LLMs, since it can be run on a single GPU. At the higher-end of the scale, our 65B-parameter model is also competitive with the best large language models such as Chinchilla or PaLM-540B.  \nUnlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g.“Books – 2TB” or“Social media conversations”) . There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.  \nIn the rest of this paper, we present an overview of the modifications we made to the transformer architecture (Vaswani et al., 2017), as well as our training method. We then report the performance of our models and compare with others LLMs on a set of standard benchmarks. Finally,","cbCaibknXpd55w2C","https://ap.wps.com/l/cbCaibknXpd55w2C","pdf",726566,27,"English","# Abstract\n# Introduction\n# Approach\n## Pre-training Data","[{\"question\":\"What is LLaMA, and what model sizes does it include?\",\"answer\":\"LLaMA is a collection of foundation language models ranging from 7B to 65B parameters.\"},{\"question\":\"How does LLaMA differ from many existing large language models in training data?\",\"answer\":\"LLaMA uses only publicly available data, aiming to be compatible with open-sourcing, unlike models that rely on non-public or undocumented datasets.\"},{\"question\":\"What performance comparisons does the abstract highlight?\",\"answer\":\"LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B.\"}]","LLaMA: Open and Efficient Foundation Language Models - Abstract | PDF",1789595601,9]