[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-125417-en":3,"doc-seo-125417-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},125417,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Scaling Data-Constrained Language Models","The document examines how scaling language models changes when training data becomes limited. It runs extensive experiments varying data repetition and compute budget, up to 900 billion training tokens and models with 9 billion parameters. Results show that with fixed compute, training up to four epochs on repeated data yields negligible loss changes versus unique data, while heavier repetition eventually makes compute returns decay to zero. The work proposes and validates compute-optimal scaling laws for repeated tokens and excess parameters, and tests mitigation methods such as adding code data or removing common filtering steps.","Scaling Data-Constrained Language Models  \nNiklas Muennighoff  \nHugging Face  \nAlexander M. Rush  \nHugging Face  \nBoaz Barak  \nHarvard University  \nTeven Le Scao  \nHugging Face  \nAleksandra Piktus  \nHugging Face  \nNouamane Tazi  \nHugging Face  \nSampo Pyysalo  \nUniversity of Turku  \nThomas Wolf  \nHugging Face  \nColin Raffel  \nHugging Face  \n[n.muennighoff@gmail.com](n.muennighoff@gmail.com)[arush@cornell.edu](arush@cornell.edu)[ ](arush@cornell.edu)[boaz@seas.harvard.edu](boaz@seas.harvard.edu)[teven.lescao@gmail.com](teven.lescao@gmail.com)[ola.piktus@gmail.com](ola.piktus@gmail.com)[ ](ola.piktus@gmail.com)[nouamane@huggingface.co](nouamane@huggingface.co)[ ](nouamane@huggingface.co)[sampo.pyysalo@gmail.com](sampo.pyysalo@gmail.com)[ ](sampo.pyysalo@gmail.com)[thomas@huggingface.co](thomas@huggingface.co)[ ](thomas@huggingface.co)[craffel@gmail.com](craffel@gmail.com)  \nEditor: Fei Sha  \nAbstract  \nThe current trend of scaling language models involves increasing both parameter count and training data set size. Extrapolating this trend suggests that training data set size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training data set with code data or removing commonly used filters. Models and data sets from our 400 training runs are freely available at [https://github.com/huggingface/datablations](https://github.com/huggingface/datablations).  \nKeywords: large language models, scaling laws, data-constrained, data engineering  \n􀀍c2025 Niklas Muennighoff and Alexander M. Rush and Boaz Barak and Teven Le Scao and Aleksandra Piktus and Nouamane Tazi and Sampo Pyysalo and Thomas Wolf and Colin Raffel.  \nLicense: CC-BY 4.0, see [https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/. Attribution)[. Attribution](https://creativecommons.org/licenses/by/4.0/. Attribution) requirements are provided  \nat [http://jmlr.org/papers/v26/24-1000.html](http://jmlr.org/papers/v26/24-1000.html).  \nMuennighoff, Rush, Barak, Scao, Piktus, Tazi, Pyysalo, Wolf, Raffel  \nData-Constrained Scaling Laws  \nFinal test loss  \n3.4  \n3.2  \n3.0  \n2.8  \n2.6  \n2.4  \n2.2  \n2.0  \nReturn on compute when repeating  \n12B  \n(1)  \n48B  \n(4)  \n120B  \n(10)  \n480B 1. 2T (40) (100)  \nTokens (Epochs)  \nParameters  \n8.67B  \n6.34B  \nAllocating compute when repeating  \n178B (7.1)  \n242B (9.7)  \nTokens (Epochs)  \nModels trained  \nLoss assuming repeated data is worth the same as new data Loss predicted by our data-constrained scaling laws  \nRegime of same compute (IsoFLOP)  \nEfficient frontier assuming repeated data is worth the same as new data Efficient frontier predicted by our data-constrained scaling laws  \nFigure 1: Return and Allocation when repeating data. Left: Loss of LLMs (4.2B parameters) scaled on repeated data decays predictably (§6) . Right: To maximize performance when repeating, our data-constrained scaling laws and empirical data suggest training smaller models for more epochs in contrast to what assuming Chinchilla scaling laws (Hoffmann et al., 2022) hold for repeated data would predict (§5) .  \n1. Introduction  \nRecent work on compute-optimal language models (Hoffmann et al., 2022) shows that many previously trained large language models (L","cbCaioHsBTkh0fKt","https://ap.wps.com/l/cbCaioHsBTkh0fKt","pdf",2178139,1,66,"English","en",105,"# Abstract\n# Introduction\n# Data-Constrained Scaling Laws\n# Return and Allocation when Repeating Data","[{\"question\":\"What trend motivates the study on data-constrained language model scaling?\",\"answer\":\"The work starts from the scaling trend that increases both parameter count and training dataset size, then asks what happens when dataset size is limited by available text on the internet.\"},{\"question\":\"How does data repetition affect loss under a fixed compute budget?\",\"answer\":\"With constrained data at fixed compute, training with up to four epochs of repeated data causes negligible loss changes compared with training on unique data.\"},{\"question\":\"What happens to the value of adding compute when repetition increases further?\",\"answer\":\"As repetition grows, the marginal value of additional compute decays and eventually approaches zero.\"}]","Scaling Data-Constrained Language Models | PDF",1785898806,166,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"scaling-data-constrained-language-models","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/scaling-data-constrained-language-models/125417/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What trend motivates the study on data-constrained language model scaling?","Question",{"text":75,"@type":76},"The work starts from the scaling trend that increases both parameter count and training dataset size, then asks what happens when dataset size is limited by available text on the internet.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does data repetition affect loss under a fixed compute budget?",{"text":80,"@type":76},"With constrained data at fixed compute, training with up to four epochs of repeated data causes negligible loss changes compared with training on unique data.",{"name":82,"@type":73,"acceptedAnswer":83},"What happens to the value of adding compute when repetition increases further?",{"text":84,"@type":76},"As repetition grows, the marginal value of additional compute decays and eventually approaches zero.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]