[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85509-en":3,"doc-seo-85509-105":29,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85509,34359740700684,"Finn","https://ap-avatar.wpscdn.com/avatar/1f400023980c374ae676?_k=1777273430885731487",8,"Research & Report","FineInstructions Scaling Synthetic Instructions to Pre Training Scale","Large language models are normally pre-trained with next-token prediction on massive unstructured text, then instruction-tuned using smaller supervised instruction–response datasets. This work proposes FineInstructions, a procedure that converts knowledge from pre-training corpora into billions of synthetic instruction–answer pairs using ~18M instruction templates derived from real user queries. Instantiated with human-written source documents, the synthetic data enables token-for-token pre-training from scratch using only the instruction-tuning objective and improves free-form response quality on standard benchmarks.","FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale  \nAjay Patel ∗  \nUniversity of Pennsylvania  \nColin Raffel  \nUniversity of Toronto Vector Institute Hugging Face  \nChris Callison-Burch  \nUniversity of Pennsylvania  \narXiv :2601 .22 146v2 [ cs .CL] 12 Jul 2026  \nAbstract  \nDue to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised “predict the next word” objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of “instructiontuning” data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts.  \nThese instruction templates are matched to and instantiated with humanwritten source documents from unstructured pre-training corpora. With“supervised” synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at [https://huggingface.co/fineinstructions](https://huggingface.co/fineinstructions).  \n1 Introduction  \nDuring self-supervised pre-training, LLMs are trained using a language modeling task like next token prediction over a large amount of text data. This stage of training is where models acquire the vast majority of their knowledge and where the majority of compute, resources, and time is spent (Raffel et al., 2020; Brown et al., 2020; Kaplan et al., 2020) . A pretrained LLM can then be adapted to have better instruction-following capabilities by being further trained on a relatively small amount of supervised instruction-answer examples in a process known as instruction-tuning (Ouyang et al., 2022; Wei et al., 2021; Sanh et al., 2021; Mishra et al., 2022) . Existing instruction-tuning datasets have various issues. Many are relatively small, consisting of a few thousand examples (Conover et al., 2023; Rajani et al., 2023) . Others are narrow and unrealistic, consisting of academic NLP tasks converted into instruction-tuning formats with a relatively small number of task templates (Sanh et al., 2021; Wei et al., 2021; Mishra et al., 2022) . Frontier language models have been used to generate large quantities of more diverse instruction-answer examples, but this has been shown to ultimately only help mimic those models superficially through distillation (Taori et al., 2023; Mukherjee et al., 2023; Honovich et al., 2022; Gudibande et al., 2023) . These issues limit the instruction-tuning stage of LLMs, making it primarily useful for helping the model learn to follow instructions and learn response styles. Consequently, the self-supervised pre-training stage is responsible for encoding the vast majority of knowledge in the model weights (Zhou et al., 2023; Ghosh et al., 2024; Hewitt et al., 2024) .  \n∗ Corresponding author: [patel.ajay285@gmail.com](patel.ajay285@gmail.com)  \n~18M Instruction Templates  \n“Between \u003Cfi>Entity A\u003C/fi> and \u003Cfi>Entity B\u003C/fi> which is more \u003Cfi>characteristic\u003C/fi>?”  \n| Pre-Training Document: |  |\n| --- | --- |\n| [GadgetBlog.com](GadgetBlog.com)[ ](GadgetBlog.com)September 30, 2023\u003Cbr>Smart Watch Reviews\u003Cbr>…We tried both the Apple and the Garmin . The Apple Watch Ultra features a new rugged design, but for extreme activities , the Garmin Fen","cbCaijicholBqgK9","https://ap.wps.com/l/cbCaijicholBqgK9","pdf",1164695,1,24,"English","en",105,"# Introduction\n## Instruction-tuning challenges and motivation\n## FineInstructions pipeline overview\n# FineInstructions dataset and training\n## Instruction templates and document instantiation\n## Controlled token-for-token experiments","[{\"question\":\"What training setup and evaluation does the paper use to validate the method?\",\"answer\":\"It runs controlled token-for-token training experiments and finds that pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on benchmarks measuring free-form response quality.\"}]",1784204079,60,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":27},"fineinstructions-scaling-synthetic-instructions-to-pre-training-scale","",{"@graph":35,"@context":77},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/fineinstructions-scaling-synthetic-instructions-to-pre-training-scale/85509/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"What training setup and evaluation does the paper use to validate the method?","Question",{"text":75,"@type":76},"It runs controlled token-for-token training experiments and finds that pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on benchmarks measuring free-form response quality.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":84},[85,89,93,97,101,106,111,114,119,122,126],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":45,"category_name":99,"show_sort_weight":28,"slug":100},5,"Comic","comic",{"id":102,"doc_module":4,"doc_module_name":45,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":45,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":112,"slug":113},30,"research-report",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":45,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":98,"slug":129},19,"General","general"]