Checkpoint

Also known as: model checkpoint

On this page 7

A checkpoint, also called a model checkpoint, is a saved snapshot of a neural network at a specific point in training, containing the model weights and, in most cases, the optimizer state, learning rate schedule position and data loader progress needed to continue. In artificial intelligence, checkpoints are the basic unit of fault tolerance, reproducibility and model release: a long training run writes them periodically so that a hardware failure costs hours rather than weeks, and so that engineers can evaluate, compare and ship intermediate versions of a model.

The word covers two related but distinct artifacts. A full training checkpoint is designed to resume training exactly where it stopped, while an inference checkpoint contains only what is required to run the model. Confusing the two is a common source of failed restarts and oversized downloads.

What a training checkpoint contains

A resumable checkpoint must capture every piece of mutable state that influences the next optimization step. The weights alone are not enough, because adaptive optimizers such as the Adam optimizer keep first and second moment estimates for every parameter, and discarding them changes the trajectory of training after a restart.

The table below lists the typical components of a full training checkpoint and what happens if each one is missing.

ComponentTypical size relative to weightsEffect if omitted
Model parameters1xNothing to resume
Optimizer stateAbout 2x for Adam in FP32Loss spike or slower convergence after restart
Master weights in FP32About 2x when training in BF16Precision loss in mixed-precision training
Scheduler and step counterNegligibleWrong learning rate after resume
Random number generator stateNegligibleDifferent dropout masks and sampling order
Data loader positionSmallRepeated or skipped training examples
Components of a resumable checkpoint

Because of the optimizer state and master weights, a full checkpoint for a model trained with Adam is often three to six times larger than the weights alone. A model with 70 billion parameters stored in BF16 needs about 140 GB for inference weights, but its complete training checkpoint can exceed one terabyte.

How checkpointing works during a training run

Checkpointing is a periodic write of training state to durable storage, triggered by a step count, a time interval or an evaluation event. The choice of interval is a trade-off between lost work after a failure and the time and storage spent writing.

In large distributed training, the model is sharded across many accelerators, so each process writes its own shard. Modern frameworks support sharded and asynchronous checkpointing: the state is first copied from accelerator memory to host memory, training resumes immediately, and a background thread uploads the shards to object storage. This reduces the pause from minutes to seconds on large clusters.

The following table compares common checkpoint scheduling strategies.

StrategyHow it is triggeredBest suited for
Fixed step intervalEvery N optimizer stepsStable runs with predictable throughput
Wall-clock intervalEvery N minutesClusters with frequent preemption
Best-metric savingWhen validation loss improvesFine-tuning with early stopping
Milestone savingAt epoch ends or data mixture changesResearch analysis and ablations
Emergency savingOn a preemption or failure signalSpot instances and shared clusters
Checkpoint scheduling strategies

A practical rule is to set the interval so that the expected lost work per failure stays small compared with the run length. On clusters with thousands of accelerators, where some component fails every few hours, intervals of 15 to 60 minutes are common.

Checkpoint formats and serialization

The file format determines loading speed, safety and portability. Older formats based on Python pickling can execute arbitrary code when loaded, which makes downloading an untrusted checkpoint a real security risk. The safetensors format stores only raw tensors and a JSON header, so it loads with memory mapping and cannot run embedded code.

FormatMain useKey property
PyTorch pickle filesResearch and full training stateFlexible but can execute code
safetensorsSharing weights for inferenceSafe, fast, memory-mapped loading
Distributed checkpoint shardsLarge sharded training runsResharding to a new cluster size
GGUFLocal quantized inferenceSingle file with metadata and quantized weights
ONNXCross-framework deploymentStores the computation graph as well
Common checkpoint formats

Teams usually keep the full training state in a sharded native format and export a separate inference checkpoint, often after quantization, for distribution.

Checkpoints in the model life cycle

Checkpoints serve far more purposes than crash recovery; they are the handoff points between training stages. Pretraining produces a base checkpoint, continued pretraining and mid-training extend it, and supervised fine-tuning, RLHF or DPO start from a chosen checkpoint and produce new ones.

  • Evaluation tracking: running benchmarks on successive checkpoints reveals learning curves and regressions.
  • Checkpoint selection: the final checkpoint is not always the best; teams often pick one by validation metrics.
  • Checkpoint averaging: averaging the weights of several late checkpoints can smooth noise and improve generalization, a simple form of model merging.
  • Rollback after instability: when a loss spike or divergence occurs, training restarts from an earlier healthy checkpoint, often with a lower learning rate or skipped data.
  • Adapter checkpoints: with LoRA or QLoRA, only small adapter weights are saved, often a few megabytes, while the base checkpoint remains unchanged.

Checkpointing should not be confused with gradient checkpointing, which is a memory-saving technique that recomputes activations during the backward pass and writes nothing to disk.

Common checkpointing mistakes

Most checkpoint failures appear only when a team actually tries to restore, which is why restore tests belong in the training pipeline. The table summarizes frequent problems.

SymptomLikely causeFix
Loss jumps after resumeOptimizer state not savedSave and restore the full optimizer state
Corrupted latest checkpointCrash during a non-atomic writeWrite to a temporary path, then rename
Storage quota exhaustedNo retention policyKeep the last few plus selected milestones
Cannot load on a new clusterShards tied to old topologyUse a reshardable distributed format
Results not reproducibleRandom and data state missingStore seeds, sampler state and configuration
Checkpoint failures and fixes

A sound policy keeps a rolling window of recent checkpoints, marks selected ones as permanent, records the exact code version and configuration with each save, and periodically verifies restores in a separate job.

A non-technical view

A checkpoint works like a save point in a video game. Training a large AI model can take weeks on thousands of machines, and something always breaks along the way. Instead of starting over, the team loads the most recent save and continues from there.

A checkpoint also lets people look back. They can test the model as it was last Tuesday, compare it with today, and choose the best version to release, not simply the newest one.

A concrete example

A research team fine-tunes a 7-billion-parameter language model on customer support transcripts using eight GPUs rented as preemptible instances. They save a full checkpoint every 30 minutes, including the Adam optimizer state, the scheduler step and the data sampler position, and they keep only the five most recent saves plus the one with the lowest validation loss.

On the second day, the cloud provider reclaims the machines without warning. The job restarts on new hardware, loads the latest checkpoint and continues with less than 30 minutes of lost work. After three epochs, evaluation shows that the checkpoint from epoch two scores better on held-out tickets than the final one, which has started to overfit. The team exports that epoch-two checkpoint to safetensors for deployment, cutting the release file to roughly 14 GB.