Continued pretraining is the practice of resuming self-supervised next-token training on an already pretrained model with a new body of unlabeled text, so that the model absorbs a domain, a language or more recent knowledge before any task-specific tuning. In artificial intelligence, it sits in the training pipeline between general pretraining and post-training, and it is also called continual pretraining or domain-adaptive pretraining.
The technique matters because building a foundation model from scratch costs millions of GPU hours, while adapting an existing checkpoint to legal contracts, biomedical literature or a low-resource language often costs a small fraction of that. The central difficulty is balance: the model must learn the new distribution without suffering catastrophic forgetting of the general skills it already has.
How continued pretraining updates a model
Continued pretraining uses the same objective as original pretraining, usually causal language modeling with cross-entropy loss over the next token. No labels or instructions are needed: the training signal comes from the raw text itself, which is why the method scales to billions of domain tokens that nobody has annotated.
The process starts from a base checkpoint, not from an instruction-tuned chat model, because chat behavior is fragile and tends to degrade under long unlabeled training. Engineers then prepare a data mixture of new domain text and a share of general text, set a modest learning rate, and train for a fixed token budget. After this stage, the adapted base model normally goes through supervised fine-tuning and preference optimization again.
The steps below summarize a typical run and what each one controls.
| Stage | What happens | Main risk if skipped |
|---|---|---|
| Checkpoint selection | Pick a base model with suitable size and tokenizer | Wasted compute on a weak starting point |
| Data curation | Deduplicate, filter and clean domain text | Model learns noise and boilerplate |
| Mixture design | Blend domain text with general replay data | Severe forgetting of general skills |
| Learning rate warmup | Raise the rate gradually from near zero | Loss spikes and unstable early steps |
| Training and evaluation | Track domain and general benchmarks together | Silent regression on general tasks |
| Post-training | Reapply supervised fine-tuning and alignment | Capable base model with poor usability |
Key hyperparameters and typical ranges
The outcome of continued pretraining depends mostly on three choices: the learning rate schedule, the token budget and the replay ratio. These interact closely, because a higher learning rate absorbs new text faster but also erases prior knowledge faster.
Research on large language models has shown that rewarming the learning rate and then decaying it again, combined with a small amount of replayed general data, can approach the quality of retraining from scratch on the combined corpus at a much lower cost. Exact values vary by model size and data, so the ranges below are starting points rather than rules.
| Parameter | Typical range | What it affects |
|---|---|---|
| Peak learning rate | 10% to 100% of the original peak | Speed of adaptation and amount of forgetting |
| Warmup length | About 0.5% to 2% of steps | Stability after resuming from a decayed checkpoint |
| Token budget | Hundreds of millions to hundreds of billions | Depth of domain or language coverage |
| Replay ratio | About 5% to 30% general data | Retention of general knowledge and reasoning |
| Context length | Same as base, or extended | Ability to handle long domain documents |
| Batch size | Similar to late pretraining | Gradient noise and hardware efficiency |
A common practical rule is to monitor general benchmarks such as reading comprehension or common-sense reasoning throughout the run. If they fall sharply while domain perplexity improves, the replay ratio or learning rate needs adjustment.
Continued pretraining compared with related methods
Continued pretraining differs from fine-tuning in its data and its goal: it teaches knowledge and language patterns from unlabeled text, whereas fine-tuning teaches behavior from labeled examples. It also differs from mid-training, which usually refers to a planned phase inside the original training run that shifts the mixture toward higher-quality or longer-context data.
The comparison below separates the methods that sit close to it in the Training topic.
| Method | Data used | Primary purpose |
|---|---|---|
| Continued pretraining | Large unlabeled domain or language text | Add knowledge and vocabulary patterns |
| Pretraining | Broad web, book and code corpora | Build general capability from random weights |
| Mid-training | Curated high-quality or long-context mixtures | Refine a model within the original run |
| Supervised fine-tuning | Labeled prompt and response pairs | Teach format and task behavior |
| LoRA | Any data, with small adapter matrices | Cheap adaptation with frozen base weights |
| RLHF and DPO | Human or AI preference comparisons | Align outputs with preferences |
Parameter-efficient methods such as LoRA can be applied to continued pretraining, but they usually absorb less new knowledge than full-parameter updates. They also forget less, which makes them attractive when the domain shift is mild and the budget is small.
When continued pretraining is the right choice
Continued pretraining is worthwhile when the target domain has a large volume of unlabeled text and vocabulary or knowledge that the base model rarely saw. It is usually unnecessary when the need is mainly about output format, tone or a narrow task, where fine-tuning or retrieval is cheaper and easier to update.
| Situation | Recommended approach | Reason |
|---|---|---|
| New language with little base coverage | Continued pretraining, often with tokenizer extension | Model must learn grammar and vocabulary |
| Specialized field with billions of tokens | Continued pretraining then fine-tuning | Knowledge lives in unlabeled text |
| Facts that change weekly | Retrieval-augmented generation | Retraining cannot keep pace with updates |
| Fixed output format or style | Supervised fine-tuning | Behavior, not knowledge, is missing |
| Small dataset under 10 million tokens | Fine-tuning or retrieval | Too little text to shift the distribution |
For new languages, teams often extend the tokenizer with frequent subwords from the target language. This shortens sequences and lowers inference cost, but the new embedding rows start untrained and need enough tokens to converge, so it is rarely worth doing for small corpora.
Limitations, risks and common mistakes
The best-known risk is catastrophic forgetting, where general reasoning, coding or safety behavior degrades as the weights move toward the new distribution. Continued pretraining also cannot guarantee factual accuracy: a model that has read a regulation may still paraphrase it incorrectly, so domain models still need evaluation and often retrieval.
- Starting from a chat model usually destroys instruction following; begin from the base checkpoint and redo post-training.
- Skipping replay data saves compute but often causes large drops on general benchmarks.
- Poor deduplication makes the model memorize repeated boilerplate, disclaimers and templates.
- Benchmark contamination occurs when domain corpora contain test questions, inflating results.
- Licensing and privacy issues arise when proprietary or personal text enters the corpus without review.
A final misconception is that more domain tokens always help. Beyond a point, returns diminish and forgetting keeps growing, so the token budget should follow evaluation curves rather than the size of the available corpus.
The idea in brief
Continued pretraining is like sending an experienced generalist back to school for a specialty. The model already knows how language works, so instead of starting over, it reads a large pile of new material, such as medical papers or texts in another language, in the same way it learned originally.
The trick is to teach the new subject without erasing the old one. Teams mix in some familiar reading and keep the learning pace moderate, so the model gains expertise while keeping its broad general skills.
A practical scenario
A legal technology team wants a model that understands contracts written in Polish. Their open 7-billion-parameter base model writes fluent English but handles Polish legal terms poorly. The team gathers about 20 billion tokens of public court rulings, statutes and anonymized contracts, deduplicates them, and blends them with 15% general multilingual text as replay.
They resume training with a peak learning rate at about one third of the original value, a short warmup and cosine decay. After the run, Polish legal perplexity falls substantially while general English benchmarks drop only slightly. The team then applies supervised fine-tuning on a few thousand reviewed question and answer pairs. The result is a clause-review assistant that uses correct terminology, which lawyers still verify before relying on any summary.