Pretraining is the initial, large-scale stage of model training in which a neural network learns general representations from vast amounts of mostly unlabeled data, using a self-supervised objective such as predicting the next token. In artificial intelligence, and in model training specifically, it produces a base model (also called a foundation model) that later stages such as fine-tuning, RLHF or DPO adapt to particular tasks and behaviors.
The term is also written as pre-training. It accounts for the overwhelming majority of the compute spent on a modern language or vision model, while the later adaptation stages are comparatively short. Almost every capability a deployed model shows is learned, at least in latent form, during pretraining.
How pretraining builds a base model
Pretraining minimizes a loss function over a corpus that is far too large to label by hand. For a decoder-only language model, the objective is next-token prediction: given a sequence of tokens, the model outputs a probability distribution for the following token, and the loss is the cross-entropy between that distribution and the token that actually appears.
The weights are updated with gradient descent, usually the AdamW optimizer, over mini-batches of sequences. Because the targets come from the data itself, no human labeling is needed, which is what allows training on trillions of tokens. To predict text well, the model must internalize grammar, facts, reasoning patterns and coding conventions, so general capability emerges as a side effect of a simple objective.
Different model families use different self-supervised objectives. The table below summarizes the main ones.
| Objective | What the model predicts | Typical model type |
|---|---|---|
| Causal language modeling | The next token in a sequence | GPT-style decoders |
| Masked language modeling | Hidden tokens from both sides | BERT-style encoders |
| Span corruption | Removed spans of text | Encoder-decoder models such as T5 |
| Contrastive learning | Which image and text belong together | CLIP-style vision-language models |
| Masked image modeling | Missing image patches | Vision transformers |
Key parameters of a pretraining run
A pretraining run is defined mainly by model size, data size and compute budget. Scaling laws describe how loss falls predictably as these grow. A widely used guideline from compute-optimal studies suggests roughly 20 training tokens per parameter, although many current models are deliberately trained far beyond that ratio because a smaller model is cheaper to serve.
The values in the following table are indicative and vary greatly between laboratories and model scales.
| Parameter | Indicative range | What it influences |
|---|---|---|
| Parameters | Hundreds of millions to trillions | Capacity to store patterns |
| Training tokens | Hundreds of billions to tens of trillions | Knowledge breadth and quality |
| Context length | About 2,000 to 8,000 tokens initially | Range of dependencies learned |
| Peak learning rate | Roughly 1e-4 to 3e-4 for large models | Speed and stability |
| Learning rate schedule | Warmup, then cosine or linear decay | Final loss and stability |
| Batch size | Millions of tokens per step | Gradient noise and throughput |
Data quality matters as much as quantity. Pretraining corpora are filtered for duplicates, spam and low-quality text, balanced across domains such as web text, books, code and scientific writing, and often refined with annealing on high-quality data near the end of the run.
Pretraining compared with later training stages
Pretraining is the foundation of a multi-stage training pipeline. The later stages use far less data and compute but decide how the model behaves, follows instructions and refuses harmful requests. Confusing these stages is a common source of misunderstanding about what a model knows and how it was shaped.
| Stage | Data | Purpose |
|---|---|---|
| Pretraining | Massive unlabeled corpus | General knowledge and skills |
| Continued pretraining | Large domain-specific corpus | Add domain or language knowledge |
| Supervised fine-tuning | Thousands to millions of curated examples | Instruction following and format |
| RLHF or DPO | Human or model preference comparisons | Helpfulness and safety |
| Distillation | Outputs of a larger teacher model | Compress capability into a smaller model |
Pretraining is the clearest case of transfer learning: the representations learned on general data transfer to many downstream tasks. Parameter-efficient methods such as LoRA and QLoRA work precisely because the base model already contains most of the necessary knowledge, so only a small adjustment of weights is required.
Infrastructure and cost
Pretraining a frontier model requires thousands of accelerators running for weeks or months, coordinated through distributed training techniques such as data, tensor and pipeline parallelism. Training is usually done in mixed precision, typically bfloat16, with periodic checkpoints so that hardware failures do not lose progress.
Costs scale with total floating-point operations, which are approximately six times the parameter count times the token count for a dense transformer. This rule of thumb explains why doubling either model size or data roughly doubles compute, and why most organizations start from an existing base model rather than pretraining their own.
Limitations and common pitfalls
Pretraining creates capability but not reliability. A base model continues text rather than answering questions, reproduces biases present in its corpus and has a knowledge cutoff at the date its data was collected. These properties are addressed, never fully removed, by later stages.
| Problem | Cause | Common remedy |
|---|---|---|
| Loss spikes | Instability at high learning rates | Restart from checkpoint, lower rate |
| Memorization | Duplicated text in the corpus | Aggressive deduplication |
| Benchmark contamination | Test data present in training data | Decontamination filters |
| Weak domain coverage | Imbalanced data mixture | Reweighting or continued pretraining |
| Harmful content learned | Unfiltered web data | Filtering plus safety training |
A frequent misconception is that fine-tuning can add large amounts of new knowledge cheaply. In practice, knowledge is mostly acquired during pretraining, while fine-tuning mostly changes style, format and behavior.
A plain-language summary
Pretraining is like giving a student a huge library and asking them to read it all, guessing each next word before turning the page. Nobody marks the answers; the text itself shows whether each guess was right.
After enough reading, the student has absorbed language, facts and ways of reasoning, but has not yet learned manners or how to answer politely. That comes later. Pretraining supplies the knowledge; later training shapes the behavior.
An example from practice
A research team wants a language model for legal documents in a less widely spoken language. Pretraining a new model from scratch would require a corpus and compute budget far beyond their means, so they take an open base model that was pretrained on several trillion tokens of general text.
They run continued pretraining on about 20 billion tokens of court decisions, statutes and contracts in the target language, with a low learning rate to avoid erasing general skills. They then apply supervised fine-tuning on a few thousand question-and-answer pairs written by lawyers. The resulting model understands legal vocabulary far better than the original, at a small fraction of the cost of full pretraining.