A data mixture (also called a data mix) is the specification of which data sources a model is trained on and in what proportions, expressed as sampling weights over domains such as web text, code, books, scientific papers, mathematics and dialogue. In Artificial intelligence, and in the Training of large language models in particular, the mixture is one of the most consequential design decisions, because it shapes what the model knows, which skills it develops and how its compute budget is spent.
A mixture is not the same as a dataset. The same collection of sources can yield very different models depending on how often each source is sampled, whether small high-quality sources are repeated across epochs and how the proportions change over the course of a training run.
How a data mixture controls what the model sees
A data mixture works by assigning each source a sampling weight, which is the probability that the next training example or sequence is drawn from that source. Weights are usually stated as shares of training tokens, so a 10% weight for code means roughly one token in ten comes from code, regardless of how large the raw code corpus is.
Because sources differ in size, the weight determines the effective number of epochs for each one. A small, carefully curated source with a high weight is repeated several times, while a huge web crawl with a moderate weight may never be seen in full. Practitioners therefore track both the target share and the resulting repetition count, since excessive repetition leads to memorization and diminishing returns.
The following table lists the main parameters that define a mixture and what each one affects.
| Parameter | What it specifies | Main effect on the model |
|---|---|---|
| Domain weight | Share of tokens drawn from each source | Balance of knowledge and skills |
| Epochs per source | How many times a source is repeated | Memorization risk versus data efficiency |
| Quality threshold | Minimum filter score for inclusion | Fluency and factual reliability |
| Schedule | How weights change over training | Which skills are reinforced late |
| Token budget | Total tokens across all sources | Overall capability at fixed model size |
Typical components of a pretraining mixture
A pretraining mixture for a general-purpose language model typically combines a large filtered web corpus with smaller, denser sources that add specific capabilities. Exact proportions vary widely between models and are often undisclosed, so published figures should be read as illustrative ranges, not fixed recipes.
The table below summarizes common domains, the share they often receive in openly documented mixtures, and the capability each one tends to support.
| Domain | Typical share range | Capability it supports |
|---|---|---|
| Filtered web text | 50-85% | Broad world knowledge and fluency |
| Source code | 5-25% | Programming, structured reasoning |
| Mathematics | 1-10% | Symbolic and quantitative reasoning |
| Books and long documents | 2-10% | Long-range coherence |
| Scientific papers | 1-5% | Technical vocabulary and precision |
| Multilingual text | 2-20% | Performance outside English |
| Synthetic data | 0-20% | Targeted skills and formats |
Domain boundaries are rarely clean. Web pages contain code and mathematics, and many teams now classify documents by topic and quality classifiers instead of by origin, which turns the mixture into a weighting over finer-grained clusters.
How teams choose mixture weights
Mixture weights are chosen through a combination of heuristics, small-scale ablations and automated optimization. Running full-size experiments for every candidate is unaffordable, so teams train proxy models with far fewer parameters and extrapolate the results, accepting that some effects do not transfer cleanly across scale.
Several systematic methods have emerged, each with different cost and assumptions.
| Approach | How it works | Main limitation |
|---|---|---|
| Manual heuristics | Experts set shares from experience | Hard to justify or reproduce |
| Ablation sweeps | Small models trained on many candidate mixes | Results may not transfer to large scale |
| DoReMi-style reweighting | A proxy model upweights domains with high excess loss | Depends on a reference model |
| Data mixing laws | Regression predicts loss from mixture proportions | Needs many runs to fit reliably |
| Online adaptive mixing | Weights adjust during training from loss signals | Adds complexity and instability risk |
A recurring finding is that validation loss alone is a weak guide. A mixture that minimizes average loss on web text can underperform on reasoning benchmarks, so teams evaluate candidate mixtures on a broad downstream evaluation suite before committing compute.
Data mixture across the training pipeline
The data mixture is not fixed for the whole lifecycle of a model; it changes deliberately at each stage. Many recipes anneal toward high-quality data near the end of pretraining, lowering the learning rate while raising the share of curated mathematics, code and instruction-like text, a phase often called mid-training.
The table below compares how mixture priorities shift between stages.
| Stage | Mixture emphasis | Main risk |
|---|---|---|
| Pretraining | Breadth and scale of diverse sources | Low-quality data wastes compute |
| Mid-training or annealing | Curated, dense, skill-specific data | Overfitting to small sources |
| Continued pretraining | New domain blended with original data | Catastrophic forgetting |
| Supervised fine-tuning | Task and instruction examples | Narrow behavior and format bias |
In continued pretraining, keeping a replay share of the original mixture, often in the range of 5-30%, is a standard defense against catastrophic forgetting. The right share depends on how far the new domain lies from the original distribution.
Mixture design is related to curriculum learning but not identical to it. A curriculum orders examples by difficulty, whereas a mixture schedule changes proportions between sources; the two are often combined in practice.
Common mistakes and limitations
Most mixture failures come from mismeasured sources or untested assumptions rather than from the weighting method itself. The following problems recur in practice.
- Ignoring repetition counts: a high weight on a small source silently means many epochs and memorized text.
- Benchmark contamination: upweighting sources that contain evaluation questions inflates scores without real gains.
- Optimizing a single metric: tuning for one benchmark often degrades general ability elsewhere.
- Assuming transfer across scale: optimal weights for a small proxy model can shift for a much larger one.
- Neglecting deduplication: near-duplicate documents distort the effective weight of a domain.
- Undocumented mixtures: without recorded proportions, results cannot be reproduced or audited.
A further limitation is that domain labels are coarse. Two web pages in the same domain can differ enormously in usefulness, which is why modern pipelines pair mixture weights with document-level quality filtering.
A non-technical view
Think of a data mixture as a recipe for a diet. A model learns from whatever it reads, and the mixture decides how much of its reading comes from websites, programs, textbooks, mathematics or conversations.
Too much of one ingredient and the model becomes lopsided. Too little and a skill never develops. Getting the proportions right matters as much as having good ingredients, and teams test many recipes on small models before cooking the full meal.
A concrete example
A research team plans to train a 7-billion-parameter model on 2 trillion tokens and wants stronger programming ability without losing general knowledge. Its starting mixture uses 80% filtered web text, 8% code, 4% mathematics, 5% books and 3% scientific papers.
The team trains a dozen small proxy models with code shares between 8% and 25%. Code benchmark scores rise steadily up to about 18%, while general knowledge scores stay flat until roughly 20% and then begin to fall. The code corpus is large enough that an 18% share means fewer than two epochs, so memorization is not a concern.
The final recipe uses 18% code and adds a short annealing phase that raises mathematics to 12% in the last 5% of training. The full-size model gains noticeably on coding tasks while matching the original mixture on general knowledge evaluations.