Mid-training is an intermediate phase of model training in which a pretrained language model continues next-token prediction on a curated data mixture before it enters post-training. In artificial intelligence, the term describes the stage that sits between broad pretraining and supervised fine-tuning, where developers shift the data toward higher quality, specific capabilities and longer contexts without changing the basic training objective.
The phase usually consumes a small fraction of total compute, often a few percent of the pretraining token budget, yet it has an outsized effect on reasoning, coding, mathematics and long-document behavior. Because it runs at the end of the learning rate schedule, mid-training also shapes how stable and receptive the final base model is when alignment work begins.
How mid-training fits into the training pipeline
Mid-training sits after the main pretraining run and before instruction tuning, and it keeps the self-supervised objective of pretraining while changing almost everything around it. The model still minimizes cross-entropy loss over raw tokens, but the data, sequence length and optimizer settings are adjusted to prepare the network for the narrower work of post-training.
The table below places mid-training among the stages that a modern large language model typically passes through.
| Stage | Main data | Objective | Typical share of tokens |
|---|---|---|---|
| Pretraining | Trillions of web, book and code tokens | Next-token prediction | About 90-99% |
| Mid-training | Filtered high-quality and domain data | Next-token prediction | About 1-10% |
| Supervised fine-tuning | Instruction and response pairs | Imitation of demonstrations | Well under 1% |
| Preference optimization | Ranked or scored responses | RLHF, DPO or similar | Well under 1% |
The boundaries are not standardized. Some laboratories call the same phase annealing, continued pretraining or a cooldown stage, and some fold it into the final segment of pretraining. What defines mid-training in practice is the deliberate change of data distribution near the end of base-model training, not a fixed recipe.
What changes during mid-training
Mid-training changes the data mixture, the context length and the learning rate while the architecture stays the same. Each lever targets a different weakness of a model that has only seen broad, noisy web text.
| Lever | Typical adjustment | Intended effect |
|---|---|---|
| Data quality | Upweight filtered educational text | Cleaner knowledge and fewer artifacts |
| Domain mix | Add mathematics, code and science | Stronger reasoning and technical skill |
| Context length | Extend from 4K-8K to 32K-128K tokens | Usable long-document attention |
| Learning rate | Decay toward a small final value | Consolidation and lower final loss |
| Synthetic data | Add generated explanations and solutions | Denser signal per token |
| Format exposure | Include dialogue and instruction-like text | Smoother handoff to fine-tuning |
Long-context extension is one of the most common mid-training goals. Developers raise the sequence length and often adjust the rotary position embedding base frequency so that attention generalizes to longer inputs. Doing this during mid-training is far cheaper than training at long sequence lengths from the start, because attention cost grows quickly with length.
Learning rate annealing is the second key mechanism. Many schedules, such as warmup-stable-decay, hold the learning rate high for most of pretraining and then decay it sharply. Placing the best data inside that decay window lets the model absorb high-value tokens while its weights settle, which tends to produce larger benchmark gains than adding the same data earlier.
Mid-training compared with neighboring concepts
Mid-training is easily confused with continued pretraining, fine-tuning and curriculum learning, because all four modify what a model sees over time. The difference lies in who runs the stage, on which data and for what purpose.
| Approach | Starting point | Data focus | Main purpose |
|---|---|---|---|
| Mid-training | Own base model near end of pretraining | Curated broad and capability data | Prepare a stronger general base model |
| Continued pretraining | Any released base model | Single domain or language | Adapt a model to a new domain |
| Supervised fine-tuning | Base or mid-trained model | Labeled demonstrations | Teach instruction following and format |
| Curriculum learning | Untrained or partly trained model | Ordered by difficulty | Improve learning efficiency throughout |
The distinction from continued pretraining is mostly one of intent. Continued pretraining usually adapts an existing model to a specific domain, such as legal or biomedical text, often by a different team. Mid-training is part of the original developer’s plan and aims to raise general capability across many tasks at once.
Designing a mid-training stage
Designing mid-training means choosing data, proportions and schedule so that new capability arrives without erasing what pretraining built. Practitioners usually test candidate mixtures with short ablation runs on smaller models before committing the full budget.
| Decision | Common choice | Reason |
|---|---|---|
| Replay of pretraining data | Keep a sizable share of general text | Limits catastrophic forgetting |
| Token budget | Tens to hundreds of billions of tokens | Enough to shift behavior without full retraining |
| Benchmark decontamination | Remove overlaps with evaluation sets | Keeps reported gains honest |
| Checkpoint selection | Compare several end points | Final loss alone can mislead |
- Mix replay data from pretraining so that general knowledge and language fluency are preserved.
- Measure with held-out evaluations that are not tied to the domains being upweighted.
- Check long-context retrieval separately, because short benchmarks do not reveal positional failures.
Limitations and common mistakes
The main risk of mid-training is distribution shift that is too abrupt. A mixture dominated by mathematics or code can raise targeted scores while degrading general writing, multilingual ability or factual recall, a form of catastrophic forgetting.
A second problem is evaluation contamination. High-quality sources such as textbooks, solution sets and synthetic explanations frequently overlap with public benchmarks, so gains may reflect memorization rather than skill. Careful teams deduplicate against test sets and report results on newer, unseen evaluations.
Finally, the term itself is loosely defined. Published descriptions differ on token counts, schedules and whether instruction-like data belongs in this stage, so comparisons between models that claim mid-training should focus on what data was actually used rather than on the label.
Stripped to the essentials
Think of a student who has read an entire public library. Before that student starts training for a specific job, a tutor hands over a shorter stack of carefully chosen books on mathematics, programming and science, plus some very long documents to practice concentration.
Mid-training is that extra reading period for an AI model. It does not teach the model to chat or follow orders yet. It simply makes the model’s general knowledge cleaner, deeper and better organized, so the later lessons on helpful behavior work better.
Where it is applied
A research team trains a 7-billion-parameter language model on about 4 trillion tokens of filtered web text with a context length of 4,096 tokens. Early evaluations show solid general knowledge but weak mathematics and poor handling of long reports.
For mid-training, the team assembles about 100 billion tokens: 40% retained general text, 25% mathematics and worked solutions, 20% source code and 15% long documents. They extend the context window to 32,768 tokens and decay the learning rate to near zero over this stage. After decontaminating the data against their test suites, they find that mathematical reasoning scores rise clearly and long-document retrieval becomes reliable, while general knowledge scores stay within a point of the earlier checkpoint. The resulting base model then goes on to supervised fine-tuning and preference optimization.