Continued pretraining

Also known as: continual pretraining

On this page 7

Continued pretraining is the practice of resuming self-supervised next-token training on an already pretrained model with a new body of unlabeled text, so that the model absorbs a domain, a language or more recent knowledge before any task-specific tuning. In artificial intelligence, it sits in the training pipeline between general pretraining and post-training, and it is also called continual pretraining or domain-adaptive pretraining.

The technique matters because building a foundation model from scratch costs millions of GPU hours, while adapting an existing checkpoint to legal contracts, biomedical literature or a low-resource language often costs a small fraction of that. The central difficulty is balance: the model must learn the new distribution without suffering catastrophic forgetting of the general skills it already has.

How continued pretraining updates a model

Continued pretraining uses the same objective as original pretraining, usually causal language modeling with cross-entropy loss over the next token. No labels or instructions are needed: the training signal comes from the raw text itself, which is why the method scales to billions of domain tokens that nobody has annotated.

The process starts from a base checkpoint, not from an instruction-tuned chat model, because chat behavior is fragile and tends to degrade under long unlabeled training. Engineers then prepare a data mixture of new domain text and a share of general text, set a modest learning rate, and train for a fixed token budget. After this stage, the adapted base model normally goes through supervised fine-tuning and preference optimization again.

The steps below summarize a typical run and what each one controls.

StageWhat happensMain risk if skipped
Checkpoint selectionPick a base model with suitable size and tokenizerWasted compute on a weak starting point
Data curationDeduplicate, filter and clean domain textModel learns noise and boilerplate
Mixture designBlend domain text with general replay dataSevere forgetting of general skills
Learning rate warmupRaise the rate gradually from near zeroLoss spikes and unstable early steps
Training and evaluationTrack domain and general benchmarks togetherSilent regression on general tasks
Post-trainingReapply supervised fine-tuning and alignmentCapable base model with poor usability
Typical stages of a continued pretraining run

Key hyperparameters and typical ranges

The outcome of continued pretraining depends mostly on three choices: the learning rate schedule, the token budget and the replay ratio. These interact closely, because a higher learning rate absorbs new text faster but also erases prior knowledge faster.

Research on large language models has shown that rewarming the learning rate and then decaying it again, combined with a small amount of replayed general data, can approach the quality of retraining from scratch on the combined corpus at a much lower cost. Exact values vary by model size and data, so the ranges below are starting points rather than rules.

ParameterTypical rangeWhat it affects
Peak learning rate10% to 100% of the original peakSpeed of adaptation and amount of forgetting
Warmup lengthAbout 0.5% to 2% of stepsStability after resuming from a decayed checkpoint
Token budgetHundreds of millions to hundreds of billionsDepth of domain or language coverage
Replay ratioAbout 5% to 30% general dataRetention of general knowledge and reasoning
Context lengthSame as base, or extendedAbility to handle long domain documents
Batch sizeSimilar to late pretrainingGradient noise and hardware efficiency
Common settings for adapting a decoder-only language model

A common practical rule is to monitor general benchmarks such as reading comprehension or common-sense reasoning throughout the run. If they fall sharply while domain perplexity improves, the replay ratio or learning rate needs adjustment.

Continued pretraining differs from fine-tuning in its data and its goal: it teaches knowledge and language patterns from unlabeled text, whereas fine-tuning teaches behavior from labeled examples. It also differs from mid-training, which usually refers to a planned phase inside the original training run that shifts the mixture toward higher-quality or longer-context data.

The comparison below separates the methods that sit close to it in the Training topic.

MethodData usedPrimary purpose
Continued pretrainingLarge unlabeled domain or language textAdd knowledge and vocabulary patterns
PretrainingBroad web, book and code corporaBuild general capability from random weights
Mid-trainingCurated high-quality or long-context mixturesRefine a model within the original run
Supervised fine-tuningLabeled prompt and response pairsTeach format and task behavior
LoRAAny data, with small adapter matricesCheap adaptation with frozen base weights
RLHF and DPOHuman or AI preference comparisonsAlign outputs with preferences
How continued pretraining relates to neighboring training methods

Parameter-efficient methods such as LoRA can be applied to continued pretraining, but they usually absorb less new knowledge than full-parameter updates. They also forget less, which makes them attractive when the domain shift is mild and the budget is small.

When continued pretraining is the right choice

Continued pretraining is worthwhile when the target domain has a large volume of unlabeled text and vocabulary or knowledge that the base model rarely saw. It is usually unnecessary when the need is mainly about output format, tone or a narrow task, where fine-tuning or retrieval is cheaper and easier to update.

SituationRecommended approachReason
New language with little base coverageContinued pretraining, often with tokenizer extensionModel must learn grammar and vocabulary
Specialized field with billions of tokensContinued pretraining then fine-tuningKnowledge lives in unlabeled text
Facts that change weeklyRetrieval-augmented generationRetraining cannot keep pace with updates
Fixed output format or styleSupervised fine-tuningBehavior, not knowledge, is missing
Small dataset under 10 million tokensFine-tuning or retrievalToo little text to shift the distribution
Decision guide for domain adaptation

For new languages, teams often extend the tokenizer with frequent subwords from the target language. This shortens sequences and lowers inference cost, but the new embedding rows start untrained and need enough tokens to converge, so it is rarely worth doing for small corpora.

Limitations, risks and common mistakes

The best-known risk is catastrophic forgetting, where general reasoning, coding or safety behavior degrades as the weights move toward the new distribution. Continued pretraining also cannot guarantee factual accuracy: a model that has read a regulation may still paraphrase it incorrectly, so domain models still need evaluation and often retrieval.

  • Starting from a chat model usually destroys instruction following; begin from the base checkpoint and redo post-training.
  • Skipping replay data saves compute but often causes large drops on general benchmarks.
  • Poor deduplication makes the model memorize repeated boilerplate, disclaimers and templates.
  • Benchmark contamination occurs when domain corpora contain test questions, inflating results.
  • Licensing and privacy issues arise when proprietary or personal text enters the corpus without review.

A final misconception is that more domain tokens always help. Beyond a point, returns diminish and forgetting keeps growing, so the token budget should follow evaluation curves rather than the size of the available corpus.

The idea in brief

Continued pretraining is like sending an experienced generalist back to school for a specialty. The model already knows how language works, so instead of starting over, it reads a large pile of new material, such as medical papers or texts in another language, in the same way it learned originally.

The trick is to teach the new subject without erasing the old one. Teams mix in some familiar reading and keep the learning pace moderate, so the model gains expertise while keeping its broad general skills.

A practical scenario

A legal technology team wants a model that understands contracts written in Polish. Their open 7-billion-parameter base model writes fluent English but handles Polish legal terms poorly. The team gathers about 20 billion tokens of public court rulings, statutes and anonymized contracts, deduplicates them, and blends them with 15% general multilingual text as replay.

They resume training with a peak learning rate at about one third of the original value, a short warmup and cosine decay. After the run, Polish legal perplexity falls substantially while general English benchmarks drop only slightly. The team then applies supervised fine-tuning on a few thousand reviewed question and answer pairs. The result is a clause-review assistant that uses correct terminology, which lawyers still verify before relying on any summary.