A checkpoint, also called a model checkpoint, is a saved snapshot of a neural network at a specific point in training, containing the model weights and, in most cases, the optimizer state, learning rate schedule position and data loader progress needed to continue. In artificial intelligence, checkpoints are the basic unit of fault tolerance, reproducibility and model release: a long training run writes them periodically so that a hardware failure costs hours rather than weeks, and so that engineers can evaluate, compare and ship intermediate versions of a model.
The word covers two related but distinct artifacts. A full training checkpoint is designed to resume training exactly where it stopped, while an inference checkpoint contains only what is required to run the model. Confusing the two is a common source of failed restarts and oversized downloads.
What a training checkpoint contains
A resumable checkpoint must capture every piece of mutable state that influences the next optimization step. The weights alone are not enough, because adaptive optimizers such as the Adam optimizer keep first and second moment estimates for every parameter, and discarding them changes the trajectory of training after a restart.
The table below lists the typical components of a full training checkpoint and what happens if each one is missing.
| Component | Typical size relative to weights | Effect if omitted |
|---|---|---|
| Model parameters | 1x | Nothing to resume |
| Optimizer state | About 2x for Adam in FP32 | Loss spike or slower convergence after restart |
| Master weights in FP32 | About 2x when training in BF16 | Precision loss in mixed-precision training |
| Scheduler and step counter | Negligible | Wrong learning rate after resume |
| Random number generator state | Negligible | Different dropout masks and sampling order |
| Data loader position | Small | Repeated or skipped training examples |
Because of the optimizer state and master weights, a full checkpoint for a model trained with Adam is often three to six times larger than the weights alone. A model with 70 billion parameters stored in BF16 needs about 140 GB for inference weights, but its complete training checkpoint can exceed one terabyte.
How checkpointing works during a training run
Checkpointing is a periodic write of training state to durable storage, triggered by a step count, a time interval or an evaluation event. The choice of interval is a trade-off between lost work after a failure and the time and storage spent writing.
In large distributed training, the model is sharded across many accelerators, so each process writes its own shard. Modern frameworks support sharded and asynchronous checkpointing: the state is first copied from accelerator memory to host memory, training resumes immediately, and a background thread uploads the shards to object storage. This reduces the pause from minutes to seconds on large clusters.
The following table compares common checkpoint scheduling strategies.
| Strategy | How it is triggered | Best suited for |
|---|---|---|
| Fixed step interval | Every N optimizer steps | Stable runs with predictable throughput |
| Wall-clock interval | Every N minutes | Clusters with frequent preemption |
| Best-metric saving | When validation loss improves | Fine-tuning with early stopping |
| Milestone saving | At epoch ends or data mixture changes | Research analysis and ablations |
| Emergency saving | On a preemption or failure signal | Spot instances and shared clusters |
A practical rule is to set the interval so that the expected lost work per failure stays small compared with the run length. On clusters with thousands of accelerators, where some component fails every few hours, intervals of 15 to 60 minutes are common.
Checkpoint formats and serialization
The file format determines loading speed, safety and portability. Older formats based on Python pickling can execute arbitrary code when loaded, which makes downloading an untrusted checkpoint a real security risk. The safetensors format stores only raw tensors and a JSON header, so it loads with memory mapping and cannot run embedded code.
| Format | Main use | Key property |
|---|---|---|
| PyTorch pickle files | Research and full training state | Flexible but can execute code |
| safetensors | Sharing weights for inference | Safe, fast, memory-mapped loading |
| Distributed checkpoint shards | Large sharded training runs | Resharding to a new cluster size |
| GGUF | Local quantized inference | Single file with metadata and quantized weights |
| ONNX | Cross-framework deployment | Stores the computation graph as well |
Teams usually keep the full training state in a sharded native format and export a separate inference checkpoint, often after quantization, for distribution.
Checkpoints in the model life cycle
Checkpoints serve far more purposes than crash recovery; they are the handoff points between training stages. Pretraining produces a base checkpoint, continued pretraining and mid-training extend it, and supervised fine-tuning, RLHF or DPO start from a chosen checkpoint and produce new ones.
- Evaluation tracking: running benchmarks on successive checkpoints reveals learning curves and regressions.
- Checkpoint selection: the final checkpoint is not always the best; teams often pick one by validation metrics.
- Checkpoint averaging: averaging the weights of several late checkpoints can smooth noise and improve generalization, a simple form of model merging.
- Rollback after instability: when a loss spike or divergence occurs, training restarts from an earlier healthy checkpoint, often with a lower learning rate or skipped data.
- Adapter checkpoints: with LoRA or QLoRA, only small adapter weights are saved, often a few megabytes, while the base checkpoint remains unchanged.
Checkpointing should not be confused with gradient checkpointing, which is a memory-saving technique that recomputes activations during the backward pass and writes nothing to disk.
Common checkpointing mistakes
Most checkpoint failures appear only when a team actually tries to restore, which is why restore tests belong in the training pipeline. The table summarizes frequent problems.
| Symptom | Likely cause | Fix |
|---|---|---|
| Loss jumps after resume | Optimizer state not saved | Save and restore the full optimizer state |
| Corrupted latest checkpoint | Crash during a non-atomic write | Write to a temporary path, then rename |
| Storage quota exhausted | No retention policy | Keep the last few plus selected milestones |
| Cannot load on a new cluster | Shards tied to old topology | Use a reshardable distributed format |
| Results not reproducible | Random and data state missing | Store seeds, sampler state and configuration |
A sound policy keeps a rolling window of recent checkpoints, marks selected ones as permanent, records the exact code version and configuration with each save, and periodically verifies restores in a separate job.
A non-technical view
A checkpoint works like a save point in a video game. Training a large AI model can take weeks on thousands of machines, and something always breaks along the way. Instead of starting over, the team loads the most recent save and continues from there.
A checkpoint also lets people look back. They can test the model as it was last Tuesday, compare it with today, and choose the best version to release, not simply the newest one.
A concrete example
A research team fine-tunes a 7-billion-parameter language model on customer support transcripts using eight GPUs rented as preemptible instances. They save a full checkpoint every 30 minutes, including the Adam optimizer state, the scheduler step and the data sampler position, and they keep only the five most recent saves plus the one with the lowest validation loss.
On the second day, the cloud provider reclaims the machines without warning. The job restarts on new hardware, loads the latest checkpoint and continues with less than 30 minutes of lost work. After three epochs, evaluation shows that the checkpoint from epoch two scores better on held-out tickets than the final one, which has started to overfit. The team exports that epoch-two checkpoint to safetensors for deployment, cutting the release file to roughly 14 GB.