Authors
Anastasiia SedovaSkyler SetoNatalie SchluterPierre Ablin
Topics
Natural Language Processing TechniquesLanguage and cultural evolutionTopic ModelingData Constraints Anastasiia Sedova, Skyler Seto, Natalie Schluter, Pierre Ablin Apple As language models scale, the amount of data they require grows – yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable target data with abundant generic data, which presents a fundamental trade-off: too little target data in the mixture underexposes the model to the target domain, while too much target data repeats the same examples excessively, yielding diminishing returns and eventual overfitting. We study this trade-off across more than 2,000 language-model training runs spanning multiple model and target dataset sizes, as well as several data types, including multilingual, domain-specific, and quality filtered mixtures. Across all settings, we find that repetition is a central driver of target-domain performance, and that mixture training tolerates much higher repetition than single-source training: scarce target corpora can be reused 15–20 times, with the optimal number of repetitions depending on the target data size, compute budget, and model scale. Next, we introduce a repetition-aware mixture scaling law that accounts for the decreasing value of repeated target tokens and the regularizing role of generic data. Optimizing the scaling law provides a principled way to compute effective mixture configurations, yielding practical mixture recommendations for pretraining under data constraints. Date: May 14, 2026Figure 1 Repetition dynamics for 143M model, 50M German tokens mixed with English, h defining the weight of German data in the training mix. (a) Repetition factor r grows with training tokens; beyond the repetition frontier, loss begins to increase. (b) German validation loss vs. training tokens. Stars indicate overfitting onset.1 Introduction Large language models (LLMs) have demonstrated remarkable performance across a wide range of language understanding, mathematics, science, and knowledge-intensive tasks (Luong et al., 2025; Woodruff et al., 2026; Singh et al., 2025; Anthropic, 2026). While much of this success can be attributed to large scale pretraining corpora exceeding trillions of tokens (Hojel et al., 2025; Li et al., 2024), many language model pretraining scenarios involve data that cannot be freely scaled including low-resource languages, specialized domains, and curated datasets, which offer far less unique data. During pretraining, this data is mixed with abundant generic data, such as pairing low-resource language text with English text, or a domain-specific math corpus with generic web text.50M 100M 500M 1BUnique target-domain data 2.0 2.2 2.4 2.6 2.8r = 20 r = 19 r = 19 r = 15Target data without repetitionsTarget data repeated 4 times Optimal repetitionsFigure 2 Best achievable German test loss by target data size, shown for the 539M model.However, mixing introduces a new trade-off. Thelimited data must be repeated for enough of the total training to provide sufficient target-domain signal, but high repetition leads to memorization andeventual overfitting. Figure 1 illustrates this challenge for a 143M model with 50M German tokensmixed with English. Higher fractions of Germandata start overfitting (Figure 1b) as the numberof repetitions crosses a frontier (Figure 1a). Thisraises the question: how do model scale, data size,and repetition jointly shape the outcome of mixturepretraining when target data in the mixture is constrained?Prior work has studied these two dimensions separately. Muennighoff et al. (2023) derive scalinglaws for data-constrained training, showing thatup to 4 repetitions over a fixed monolithic datasetare nearly as valuable as unique data. However,this result, widely adopted as a practical ceiling for data repetition, applies to single-source training whereall tokens are repeated. In mixture pretraining, only the constrained domain is repeated while the genericdomain continues to supply fresh tokens, which is a structurally different regime. Conversely, data mixinglaws (Ye et al., 2024; Xie et al., 2024; Shukor et al., 2025) optimize the composition of training mixturesto maximize downstream performance, but assume each data source has unlimited data. Our work sits atthe intersection: we study how to optimally mix data-constrained sources with abundant ones, jointly optimizing mixture weights and repetition, a setting that arises in many real-world pretraining scenarios. Ourcontributions are:• A systematic empirical study of over 2000 training runs spanning model sizes from 101M to 805M pa rameters, different target data sizes, and diverse data types: multilingual (German, French, Swahili), multi-domain (mathematics, scientific papers, Wikipedia), and quality-filtered subsets.• Empirical findings on repetition in data mixtures: repeated tokens follow diminishing returns predictably across all data types; optimal repetition scales with target dataset size and compute budget; and larger models consistently extract more from limited data despite overfitting faster. Crucially, abundant generic data sustains learning and unlocks far higher repetition than reported for single-source training without performance degradation, with optimal for target task performance repetitions reaching 15–20 times (Figure 2).• A scaling law at the intersection of data-constrained training and mixture optimization that predicts target-domain loss as a function of target data size, mixture ratio, and model size. We demonstrate that this scaling law can be fitted at small scales and then extrapolates at larger scales, and that it can be used to estimate accurately the optimal number of repetitions. We also extend our findings to the case of two scarce target domains. Together, our findings provide both empirical understanding of how repetition behaves in data mixtures and a predictive scaling law for training effectively on limited target data.