Data mixture

Also known as: data mix

On this page 7

A data mixture (also called a data mix) is the specification of which data sources a model is trained on and in what proportions, expressed as sampling weights over domains such as web text, code, books, scientific papers, mathematics and dialogue. In Artificial intelligence, and in the Training of large language models in particular, the mixture is one of the most consequential design decisions, because it shapes what the model knows, which skills it develops and how its compute budget is spent.

A mixture is not the same as a dataset. The same collection of sources can yield very different models depending on how often each source is sampled, whether small high-quality sources are repeated across epochs and how the proportions change over the course of a training run.

How a data mixture controls what the model sees

A data mixture works by assigning each source a sampling weight, which is the probability that the next training example or sequence is drawn from that source. Weights are usually stated as shares of training tokens, so a 10% weight for code means roughly one token in ten comes from code, regardless of how large the raw code corpus is.

Because sources differ in size, the weight determines the effective number of epochs for each one. A small, carefully curated source with a high weight is repeated several times, while a huge web crawl with a moderate weight may never be seen in full. Practitioners therefore track both the target share and the resulting repetition count, since excessive repetition leads to memorization and diminishing returns.

The following table lists the main parameters that define a mixture and what each one affects.

ParameterWhat it specifiesMain effect on the model
Domain weightShare of tokens drawn from each sourceBalance of knowledge and skills
Epochs per sourceHow many times a source is repeatedMemorization risk versus data efficiency
Quality thresholdMinimum filter score for inclusionFluency and factual reliability
ScheduleHow weights change over trainingWhich skills are reinforced late
Token budgetTotal tokens across all sourcesOverall capability at fixed model size
Core parameters of a data mixture

Typical components of a pretraining mixture

A pretraining mixture for a general-purpose language model typically combines a large filtered web corpus with smaller, denser sources that add specific capabilities. Exact proportions vary widely between models and are often undisclosed, so published figures should be read as illustrative ranges, not fixed recipes.

The table below summarizes common domains, the share they often receive in openly documented mixtures, and the capability each one tends to support.

DomainTypical share rangeCapability it supports
Filtered web text50-85%Broad world knowledge and fluency
Source code5-25%Programming, structured reasoning
Mathematics1-10%Symbolic and quantitative reasoning
Books and long documents2-10%Long-range coherence
Scientific papers1-5%Technical vocabulary and precision
Multilingual text2-20%Performance outside English
Synthetic data0-20%Targeted skills and formats
Common domains and their role in the mixture

Domain boundaries are rarely clean. Web pages contain code and mathematics, and many teams now classify documents by topic and quality classifiers instead of by origin, which turns the mixture into a weighting over finer-grained clusters.

How teams choose mixture weights

Mixture weights are chosen through a combination of heuristics, small-scale ablations and automated optimization. Running full-size experiments for every candidate is unaffordable, so teams train proxy models with far fewer parameters and extrapolate the results, accepting that some effects do not transfer cleanly across scale.

Several systematic methods have emerged, each with different cost and assumptions.

ApproachHow it worksMain limitation
Manual heuristicsExperts set shares from experienceHard to justify or reproduce
Ablation sweepsSmall models trained on many candidate mixesResults may not transfer to large scale
DoReMi-style reweightingA proxy model upweights domains with high excess lossDepends on a reference model
Data mixing lawsRegression predicts loss from mixture proportionsNeeds many runs to fit reliably
Online adaptive mixingWeights adjust during training from loss signalsAdds complexity and instability risk
Approaches to setting mixture weights

A recurring finding is that validation loss alone is a weak guide. A mixture that minimizes average loss on web text can underperform on reasoning benchmarks, so teams evaluate candidate mixtures on a broad downstream evaluation suite before committing compute.

Data mixture across the training pipeline

The data mixture is not fixed for the whole lifecycle of a model; it changes deliberately at each stage. Many recipes anneal toward high-quality data near the end of pretraining, lowering the learning rate while raising the share of curated mathematics, code and instruction-like text, a phase often called mid-training.

The table below compares how mixture priorities shift between stages.

StageMixture emphasisMain risk
PretrainingBreadth and scale of diverse sourcesLow-quality data wastes compute
Mid-training or annealingCurated, dense, skill-specific dataOverfitting to small sources
Continued pretrainingNew domain blended with original dataCatastrophic forgetting
Supervised fine-tuningTask and instruction examplesNarrow behavior and format bias
Mixture priorities by training stage

In continued pretraining, keeping a replay share of the original mixture, often in the range of 5-30%, is a standard defense against catastrophic forgetting. The right share depends on how far the new domain lies from the original distribution.

Mixture design is related to curriculum learning but not identical to it. A curriculum orders examples by difficulty, whereas a mixture schedule changes proportions between sources; the two are often combined in practice.

Common mistakes and limitations

Most mixture failures come from mismeasured sources or untested assumptions rather than from the weighting method itself. The following problems recur in practice.

  • Ignoring repetition counts: a high weight on a small source silently means many epochs and memorized text.
  • Benchmark contamination: upweighting sources that contain evaluation questions inflates scores without real gains.
  • Optimizing a single metric: tuning for one benchmark often degrades general ability elsewhere.
  • Assuming transfer across scale: optimal weights for a small proxy model can shift for a much larger one.
  • Neglecting deduplication: near-duplicate documents distort the effective weight of a domain.
  • Undocumented mixtures: without recorded proportions, results cannot be reproduced or audited.

A further limitation is that domain labels are coarse. Two web pages in the same domain can differ enormously in usefulness, which is why modern pipelines pair mixture weights with document-level quality filtering.

A non-technical view

Think of a data mixture as a recipe for a diet. A model learns from whatever it reads, and the mixture decides how much of its reading comes from websites, programs, textbooks, mathematics or conversations.

Too much of one ingredient and the model becomes lopsided. Too little and a skill never develops. Getting the proportions right matters as much as having good ingredients, and teams test many recipes on small models before cooking the full meal.

A concrete example

A research team plans to train a 7-billion-parameter model on 2 trillion tokens and wants stronger programming ability without losing general knowledge. Its starting mixture uses 80% filtered web text, 8% code, 4% mathematics, 5% books and 3% scientific papers.

The team trains a dozen small proxy models with code shares between 8% and 25%. Code benchmark scores rise steadily up to about 18%, while general knowledge scores stay flat until roughly 20% and then begin to fall. The code corpus is large enough that an 18% share means fewer than two epochs, so memorization is not a concern.

The final recipe uses 18% code and adds a short annealing phase that raises mathematics to 12% in the last 5% of training. The full-size model gains noticeably on coding tasks while matching the original mixture on general knowledge evaluations.