Demystifying Synthetic Data in LLM Pre-training - A Systematic Study of Scaling Laws, Benefits, and Pitfalls

Oct 12, 2026

Uploaded by nayy☆

Synthetic data techniques offer a way to mitigate the limited availability of high-quality text for large language model pre-training. This study performs a large-scale empirical investigation (>1000 LLM variants, >100k GPU hours) with unified protocols and scaling-law analysis. Results compare natural web data, diverse synthetic types, and their mixtures. Rephrased synthetic data alone does not speed up pre-training, while mixing about one third rephrased synthetic with two thirds natural web text yields 5–10x faster convergence to the same validation loss. Textbook-style generated data alone increases loss, especially under small data budgets.

LanguageEnglish

CategoryGeneral

FormatPDF

Pages20

Reading Time7 min

AccessFree to download

Abstract

Introduction

Research Questions (RQ1–RQ3)

Motivation and Related Challenges

Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls Feiyang Kang1,2,* , Newsha Ardalani1 , Michael Kuchnik1 , Youssef Emad1 , Mostafa Elhoushi3,* , Shubhabrata Sengupta4,* , Shang-Wen Li1 , Ramya Raghavendra1 , Ruoxi Jia2 , Carole-Jean Wu1 1FAIR at Meta, 2Virginia Tech, 3 Cerebras Systems, 4Independent consultant *Work done at Meta. Correspondence: Feiyang Kang <[fyk@vt.edu](fyk@vt.edu)>, Newsha Ardalani <[new@meta.com](new@meta.com)> Abstract Training data plays a crucial role in Large Language Models (LLM) scaling, yet high quality data is of limited supply. Synthetic data techniques offer a potential path toward sidestepping these limitations.We conduct a large-scale empirical investigation (>1000 LLMs with >100k GPU hours) using a unified protocol and scaling laws, comparing natural web data, diverse synthetic types (rephrased text, generated textbooks), and mixtures of natural and synthetic data. Specifically, we found pre-training on rephrased synthetic data alone is not faster than pre-training on natural web texts; while pre-training on 1/3 rephrased synthetic data mixed with 2/3 natural web texts can speedup 5-10x (to reach the same validation loss) at larger data budgets. Pre-training on textbookstyle synthetic data alone results in notably higher loss on many downstream domains especially at small data budgets.“Good” ratios of synthetic data in training data mixtures depend on the model size and data budget, empirically converging to ∼30% for rephrased synthetic data. Larger generator models do not necessarily yield better pre-training data than ∼8Bparam models. These results contribute mixed evidence on “model collapse" during largescale single-round (n=1) model training on synthetic data–training on rephrased synthetic data shows no degradation in performance in foreseeable scales whereas training on mixtures of textbook-style pure-generated synthetic data shows patterns predicted by “model collapse". Our work demystifies synthetic data in pretraining, validates its conditional benefits, and offers practical guidance. 1 Introduction The remarkable advancements in Large Language Models (LLMs) are closely tied to the scale and, critically, the quality of their training data. As computational demands for training state-of-theart models escalate and the finite nature of high- quality natural text becomes increasingly apparent (Villalobos et al., 2024), significant interest has turned towards synthetic data (Ben Allal et al., 2024 ; Eldan and Li, 2023 ; Patel et al., 2024 ; Chenet al., 2024 ; Long et al., 2024 ; Thrush et al., 2024 ; Havrilla et al., 2024 ; Maini et al., 2024 ; Li et al., 2023b ; Abdin et al., 2024 ; Javaheripi et al., 2023 ; Cheng et al., 2024 ; Gu et al., 2023) . Defined as text generated by pre-existing models or automated pipelines, synthetic data presents a compelling potential avenue for augmenting—or perhaps eventually replacing—traditional human-generated corpora during the foundational pre-training phase. While the utility of synthetic data is increasingly established in post-training stages like instructiontuning and alignment (Taori et al., 2023 ; Li et al., 2023a ; Ge et al., 2024)—where objectives are targeted and natural data may be scarce—its role and effects during the crucial pre-training phase remain largely uncharacterized and poorly understood (Liu et al., 2024b) . This knowledge gap represents a significant barrier to optimizing LLM development pipelines and motivates fundamental questions: (RQ1) Can synthetic data effectively enhance LLMpre-training performance at large data scalesand under what conditions? (RQ2) How do different types and generation methodologies for synthetic data influence pre-training dynamics and scaling behavior? (RQ3) What principles guide the effective deployment of synthetic data in pre-training, including “good” mixture ratios, the impact of generator model capabilities, an

FAQs

What does the study investigate about synthetic data in LLM pre-training?

It examines whether synthetic data improves pre-training at large data scales and under what conditions, how different synthetic types and generation methods affect training dynamics, and what principles govern effective deployment such as mixture ratios and generator model effects.

What happens when pre-training uses only rephrased synthetic data?

Pre-training on rephrased synthetic data alone is not faster than pre-training on natural web texts, according to the reported comparisons and convergence behavior.

How can synthetic data mixtures improve training efficiency?

A mixture of about 1/3 rephrased synthetic data with 2/3 natural web text can speed up pre-training by 5–10x to reach the same validation loss at larger data budgets.

Oct 12, 2026

Uploaded by nayy☆