Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

Jul 16, 2026

Upload by Noah

Large language model training has shifted toward multi-epoch optimization, where reusing limited high-quality data can improve performance and sample efficiency. Excessive repetition, however, increases overfitting risk and yields diminishing returns, leaving open how and when to reuse data effectively. Using memorization-window signals from loss retention dynamics and downstream evaluation, the approach adaptively schedules data replays and epoch counts. Preliminary experiments show repetition can improve beyond commonly cited limits, supporting memorization-aware training budgets.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

Introduction

Validation loss and epoch-size effects

Marginal returns of repetition vs. fresh data

Memorization-guided data reuse

Contributions

arXiv :2607 .04969v 1 [ cs .LG] 6 Jul 2026 TRAIN SMARTER, NOT LONGER: MEMORIZATIONGUIDED DATA REUSE FOR EFFICIENT LLM TRAINING Jingwei Zuo*, Cong Zeng, Ilyas Chahed, Maksim Velikanov, Dhia Eddine Rhaiem, Pasquale Balsebre, Abhay Kumar, Younes Belkada, Hakim Hacid Technology Innovation Institute Abu Dhabi, UAE ABSTRACT The training paradigm of large language models has shifted from traditional onepass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency. Meanwhile, excessive repetition introduces the risk of overfitting and diminishing returns. Determining when and how to reuse data effectively thus emerges as a natural but underexplored question. Through a novel observation of model’s Memorization Window signals derived from loss retention dynamics and downstream evaluation scores, we propose Memorization-guided Data Reuse, a training paradigm that adaptively determines when and how data should be reused, enabling principled decisions on the number of training epochs and the scheduling of data replays. Our preliminary experiments reveal a consistent memorization-driven regime: performance continues to improve with repetition far beyond current practice (e.g., the commonly cited four-epoch limit) . While a full scheduler remains future work, these insights provide a foundation for memorization-aware training schedules, helping to determine reuse budgets and move toward training LLMs smarter rather than longer with limited high-quality data. 1 INTRODUCTION Va l idat ion Loss 4.0 3.8 3.6 3.4 3.2 3.0 2.8 2.6 Validation Loss during multi-epoch training 0 100 200 300 400 500 Gigatokens (a) Epoch size effect on validation loss Rescaled Va l idation Loss 0.6 0.5 0.4 0.3 0.2 0.1 0.0 Repetition returns compared to single epoch training 0 100 200 300 400 500 Gigatokens (b) Marginal Returns vs. Fresh Data Figure 1: Epoch size determines the value of repetition. Under memorization-aware data scheduler (100M Transformer): (a) Validation loss for varying epoch sizes N. Small N leads to rapid overfitting, while larger N tracks the infinite-data baseline. (b) Returns from repetition vs. fresh data. When spacing is sufficient (e.g., N = 4 .0GT), the loss decreases at the same rate as the full dataset (brown), implying that re-learning forgotten data yields the same marginal gain as injecting new data Despite most LLMs were trained with a single epoch prior to 2024, recent advances in large language model (LLM) training rely heavily on multi-epoch training, in which training data are reused across *Correspondence to: [jingwei.zuo@tii.ae](jingwei.zuo@tii.ae) epochs to improve model performance. This practice is driven by the limited availability of highquality (HQ) data as LLMs continue to scale. Many works (Taylor et al., 2022; Lozhkov et al.; Olmo et al., 2025; Hao et al., 2025) have explored repetition strategies, typically including the reuse of high-quality (HQ) data uniformly across epochs. Since prior work has explored the effect of data repetition on training (Muennighoff et al., 2023), showing that training with up to four epochs of repeated data yields negligible changes to loss while further repetition leads to diminishing returns, most existing repetition strategies adopt a relatively conservative approach. However, we argue that they either rely on fixed heuristics for data reuse (e.g., globally shuffling multi-epoch data) or ignore the dynamics of model memorization. We illustrate this dynamic in Figure 1a by varying different epoch size, the onset of overfitting is not fixed but shifts according to the epoch size N , effectively delaying the degradation point. This leads to open questions about given limited HQ data, how to optimally schedule or reuse them during training. In practice, training with limited HQ data presents a trade-off: increasing the number of epochs on

FAQs

Why does multi-epoch LLM training increase the need for careful data reuse?

Multi-epoch training reuses limited high-quality data to improve performance and sample efficiency, but too much repetition can cause overfitting and diminishing returns.

What problem does memorization-guided data reuse address?

It determines when and how data should be reused by using memorization-related signals, enabling principled choices for the number of training epochs and the scheduling of data replays.

What key finding is reported about repeating high-quality data?

Under memorization-aware scheduling with sufficient spacing, repeated tokens can restore marginal learning gains to match fresh-data learning efficiency, even though overall performance is still bounded by limited data diversity.

Jul 16, 2026

Upload by Noah