Post-training is the set of training stages applied to a model after pretraining is complete, with the goal of turning a general next-token predictor into a system that follows instructions, reasons reliably and behaves safely. Artificial intelligence practitioners use the term mainly for large language models, where post-training covers supervised fine-tuning, preference optimization such as RLHF or DPO, reinforcement learning on verifiable tasks and safety tuning.
Pretraining gives a model broad knowledge and linguistic competence, but it does not teach the model what kind of answer people want. Post-training uses far less compute than pretraining, often a few percent of the total budget, yet it determines most of the behavior users actually experience: tone, format, refusal policy, tool use and reasoning style.
How post-training reshapes a pretrained model
Post-training works by adjusting the weights of a pretrained checkpoint on targeted data and objectives, so that the probability mass shifts toward desirable outputs. A base model already contains the capabilities needed for good answers; post-training elicits and stabilizes those capabilities rather than adding large amounts of new knowledge.
A typical pipeline runs in sequence. Supervised fine-tuning (SFT) first trains the model on curated prompt and response pairs, teaching the dialogue format and basic instruction following. Preference optimization then uses comparisons between better and worse answers to refine quality. Many modern pipelines add reinforcement learning with verifiable rewards, where math answers or unit tests provide an automatic signal, and finish with safety and policy tuning.
The table below lists the main stages, their data and their typical effect on the model.
| Stage | Training signal | Main effect |
|---|---|---|
| Supervised fine-tuning | Demonstrations written or curated by humans and models | Chat format, instruction following |
| Preference optimization | Pairwise comparisons or reward model scores | Helpfulness, style, fewer poor answers |
| Reinforcement learning on verifiable tasks | Automatic checks such as test results or exact answers | Longer, more accurate reasoning |
| Safety tuning | Policy examples, red-team prompts, constitutional principles | Appropriate refusals, harm reduction |
| Distillation | Outputs of a stronger teacher model | Smaller model inherits behavior |
Key methods and objectives in post-training
Post-training methods differ mainly in where the learning signal comes from and how much infrastructure they need. SFT uses a standard cross-entropy loss on target responses. RLHF trains a separate reward model on human preference data and optimizes the policy against it, usually with PPO and a KL penalty that keeps the model close to its reference. DPO removes the explicit reward model and optimizes the preference objective directly.
The following comparison shows the practical trade-offs between the most widely used methods.
| Method | Data required | Compute and complexity | Typical risk |
|---|---|---|---|
| SFT | Thousands to millions of demonstrations | Low, single model | Imitates flaws in demonstrations |
| RLHF with PPO | Preference pairs plus reward model | High, several models in memory | Reward hacking, instability |
| DPO | Preference pairs only | Moderate, offline training | Overfitting to preference noise |
| RLAIF | Preferences labeled by an AI judge | Moderate to high | Inherits judge biases |
| Rejection sampling fine-tuning | Best-of-n samples filtered by a scorer | Moderate | Narrow diversity of answers |
Parameter-efficient techniques such as LoRA and QLoRA can implement several of these stages by training small adapter matrices instead of all weights. They reduce memory needs sharply, although frontier laboratories generally run full-parameter post-training for their flagship models.
Post-training compared with pretraining and mid-training
Post-training differs from earlier stages in data volume, objective and purpose. Pretraining consumes trillions of tokens of broad text to build general competence. Mid-training or continued pretraining sits between the two and shifts the data mixture toward specific domains, long contexts or code. Post-training uses comparatively small, highly curated datasets aimed at behavior.
| Phase | Typical data scale | Objective | What it changes |
|---|---|---|---|
| Pretraining | Trillions of tokens | Next-token prediction on broad text | Knowledge and general skills |
| Mid-training | Billions to hundreds of billions of tokens | Next-token prediction on targeted mixtures | Domain depth, context length |
| Post-training | Thousands to millions of examples | Imitation, preference and reward objectives | Behavior, format, alignment |
The boundaries are not strict. Some laboratories mix instruction data into late pretraining, and the term fine-tuning is sometimes used loosely for any post-training step. In precise usage, fine-tuning is one technique within post-training, while post-training names the whole stage.
Designing a post-training run in practice
A well-designed post-training run starts from a clear behavioral specification: which tasks matter, which formats are expected and which requests must be refused. Data quality outweighs data quantity at this stage, and a few thousand carefully reviewed examples can outperform a much larger noisy set for SFT.
Teams usually tune a small set of parameters that strongly influence the outcome. The table gives typical starting ranges, which vary with model size and method.
| Setting | Typical range | What it affects |
|---|---|---|
| Learning rate for SFT | About 1e-6 to 2e-5 for full fine-tuning | Speed of adaptation and forgetting |
| Epochs over SFT data | 1 to 3 | Memorization versus generalization |
| KL coefficient or DPO beta | Roughly 0.01 to 0.5 | Distance from the reference model |
| Share of safety data | A few percent of the mixture | Balance of refusals and helpfulness |
Evaluation runs alongside training. Teams track benchmark scores, win rates against earlier checkpoints and targeted regression tests, because improvements on one behavior often degrade another. Model merging of several post-trained checkpoints is a common final step to combine strengths.
Limitations and failure modes of post-training
Post-training can only shape what the base model already supports. It rarely adds deep new knowledge, and attempts to teach many new facts through SFT tend to increase hallucination rather than accuracy. Several other failure modes are well documented:
- Catastrophic forgetting: aggressive tuning erodes capabilities learned in pretraining.
- Reward hacking: the policy exploits weaknesses of the reward model, for example by producing long but empty answers.
- Sycophancy: preference data from human raters rewards agreement and flattery over accuracy.
- Over-refusal: excessive safety tuning causes the model to decline harmless requests.
- Alignment tax: some benchmark scores drop slightly after alignment-focused stages.
Most of these problems are mitigated by regularization toward the reference model, diverse and balanced data, and continuous evaluation. None of them is fully solved, which is why post-training remains an active and iterative engineering discipline.
In plain words
Imagine someone who has read an entire library but has never held a conversation. They know a great deal, yet they ramble, ignore questions and sometimes say inappropriate things. Pretraining produces that kind of model.
Post-training is the coaching that comes afterward. The model sees examples of good answers, learns which of two answers people prefer and practices tasks where correct results can be checked. The result is a model that answers clearly, follows instructions and knows when to say no. The knowledge came from reading; the manners and judgment come from post-training.
A real-world example
A software company has an open-weight base model with 8 billion parameters and wants an internal assistant for its support engineers. The base model writes plausible text but often continues the prompt instead of answering it and ignores the company’s required answer format.
The team first runs SFT on about 12,000 reviewed support conversations for two epochs. It then collects roughly 20,000 preference pairs, where engineers choose the better of two model answers, and trains with DPO at a beta of 0.1. A small safety set teaches the model to decline requests for customer credentials.
After post-training, internal reviewers prefer the new assistant over the SFT-only version in about two thirds of blind comparisons, and format errors fall sharply. General reasoning benchmarks stay within one point of the base model, which shows that forgetting was kept under control.