Mid-training

On this page 7

Mid-training is an intermediate phase of model training in which a pretrained language model continues next-token prediction on a curated data mixture before it enters post-training. In artificial intelligence, the term describes the stage that sits between broad pretraining and supervised fine-tuning, where developers shift the data toward higher quality, specific capabilities and longer contexts without changing the basic training objective.

The phase usually consumes a small fraction of total compute, often a few percent of the pretraining token budget, yet it has an outsized effect on reasoning, coding, mathematics and long-document behavior. Because it runs at the end of the learning rate schedule, mid-training also shapes how stable and receptive the final base model is when alignment work begins.

How mid-training fits into the training pipeline

Mid-training sits after the main pretraining run and before instruction tuning, and it keeps the self-supervised objective of pretraining while changing almost everything around it. The model still minimizes cross-entropy loss over raw tokens, but the data, sequence length and optimizer settings are adjusted to prepare the network for the narrower work of post-training.

The table below places mid-training among the stages that a modern large language model typically passes through.

StageMain dataObjectiveTypical share of tokens
PretrainingTrillions of web, book and code tokensNext-token predictionAbout 90-99%
Mid-trainingFiltered high-quality and domain dataNext-token predictionAbout 1-10%
Supervised fine-tuningInstruction and response pairsImitation of demonstrationsWell under 1%
Preference optimizationRanked or scored responsesRLHF, DPO or similarWell under 1%
Stages of a typical language model training pipeline

The boundaries are not standardized. Some laboratories call the same phase annealing, continued pretraining or a cooldown stage, and some fold it into the final segment of pretraining. What defines mid-training in practice is the deliberate change of data distribution near the end of base-model training, not a fixed recipe.

What changes during mid-training

Mid-training changes the data mixture, the context length and the learning rate while the architecture stays the same. Each lever targets a different weakness of a model that has only seen broad, noisy web text.

LeverTypical adjustmentIntended effect
Data qualityUpweight filtered educational textCleaner knowledge and fewer artifacts
Domain mixAdd mathematics, code and scienceStronger reasoning and technical skill
Context lengthExtend from 4K-8K to 32K-128K tokensUsable long-document attention
Learning rateDecay toward a small final valueConsolidation and lower final loss
Synthetic dataAdd generated explanations and solutionsDenser signal per token
Format exposureInclude dialogue and instruction-like textSmoother handoff to fine-tuning
Main levers adjusted in mid-training

Long-context extension is one of the most common mid-training goals. Developers raise the sequence length and often adjust the rotary position embedding base frequency so that attention generalizes to longer inputs. Doing this during mid-training is far cheaper than training at long sequence lengths from the start, because attention cost grows quickly with length.

Learning rate annealing is the second key mechanism. Many schedules, such as warmup-stable-decay, hold the learning rate high for most of pretraining and then decay it sharply. Placing the best data inside that decay window lets the model absorb high-value tokens while its weights settle, which tends to produce larger benchmark gains than adding the same data earlier.

Mid-training compared with neighboring concepts

Mid-training is easily confused with continued pretraining, fine-tuning and curriculum learning, because all four modify what a model sees over time. The difference lies in who runs the stage, on which data and for what purpose.

ApproachStarting pointData focusMain purpose
Mid-trainingOwn base model near end of pretrainingCurated broad and capability dataPrepare a stronger general base model
Continued pretrainingAny released base modelSingle domain or languageAdapt a model to a new domain
Supervised fine-tuningBase or mid-trained modelLabeled demonstrationsTeach instruction following and format
Curriculum learningUntrained or partly trained modelOrdered by difficultyImprove learning efficiency throughout
How mid-training differs from related training approaches

The distinction from continued pretraining is mostly one of intent. Continued pretraining usually adapts an existing model to a specific domain, such as legal or biomedical text, often by a different team. Mid-training is part of the original developer’s plan and aims to raise general capability across many tasks at once.

Designing a mid-training stage

Designing mid-training means choosing data, proportions and schedule so that new capability arrives without erasing what pretraining built. Practitioners usually test candidate mixtures with short ablation runs on smaller models before committing the full budget.

DecisionCommon choiceReason
Replay of pretraining dataKeep a sizable share of general textLimits catastrophic forgetting
Token budgetTens to hundreds of billions of tokensEnough to shift behavior without full retraining
Benchmark decontaminationRemove overlaps with evaluation setsKeeps reported gains honest
Checkpoint selectionCompare several end pointsFinal loss alone can mislead
Design decisions and common choices
  • Mix replay data from pretraining so that general knowledge and language fluency are preserved.
  • Measure with held-out evaluations that are not tied to the domains being upweighted.
  • Check long-context retrieval separately, because short benchmarks do not reveal positional failures.

Limitations and common mistakes

The main risk of mid-training is distribution shift that is too abrupt. A mixture dominated by mathematics or code can raise targeted scores while degrading general writing, multilingual ability or factual recall, a form of catastrophic forgetting.

A second problem is evaluation contamination. High-quality sources such as textbooks, solution sets and synthetic explanations frequently overlap with public benchmarks, so gains may reflect memorization rather than skill. Careful teams deduplicate against test sets and report results on newer, unseen evaluations.

Finally, the term itself is loosely defined. Published descriptions differ on token counts, schedules and whether instruction-like data belongs in this stage, so comparisons between models that claim mid-training should focus on what data was actually used rather than on the label.

Stripped to the essentials

Think of a student who has read an entire public library. Before that student starts training for a specific job, a tutor hands over a shorter stack of carefully chosen books on mathematics, programming and science, plus some very long documents to practice concentration.

Mid-training is that extra reading period for an AI model. It does not teach the model to chat or follow orders yet. It simply makes the model’s general knowledge cleaner, deeper and better organized, so the later lessons on helpful behavior work better.

Where it is applied

A research team trains a 7-billion-parameter language model on about 4 trillion tokens of filtered web text with a context length of 4,096 tokens. Early evaluations show solid general knowledge but weak mathematics and poor handling of long reports.

For mid-training, the team assembles about 100 billion tokens: 40% retained general text, 25% mathematics and worked solutions, 20% source code and 15% long documents. They extend the context window to 32,768 tokens and decay the learning rate to near zero over this stage. After decontaminating the data against their test suites, they find that mathematical reasoning scores rise clearly and long-document retrieval becomes reliable, while general knowledge scores stay within a point of the earlier checkpoint. The resulting base model then goes on to supervised fine-tuning and preference optimization.