Gradient checkpointing

Also known as: activation checkpointing

On this page 7

Gradient checkpointing, also called activation checkpointing, is a memory-saving technique in neural network training that stores only selected intermediate activations during the forward pass and recomputes the missing ones during the backward pass. In artificial intelligence, it is one of the standard tools for fitting large models, long sequences or larger batches into limited accelerator memory.

The technique makes a deliberate trade of compute for memory. A typical configuration adds roughly 20 to 35 percent more training time in exchange for reducing activation memory by a large factor, often enough to turn an out-of-memory failure into a working training run on the same hardware.

How gradient checkpointing reduces activation memory

Gradient checkpointing works by discarding most forward activations and keeping only a few checkpoints from which the rest can be rebuilt. Backpropagation needs the activations of every layer to compute gradients, so a standard implementation keeps all of them in memory until the backward pass consumes them.

For deep networks, these stored activations often dominate memory use, exceeding the size of the weights, gradients and optimizer states combined. Activation memory grows with batch size, sequence length, hidden width and depth, which is why long-context language models hit memory limits so quickly.

With checkpointing enabled, the framework divides the network into segments. During the forward pass it saves only the inputs to each segment. During the backward pass it reruns the forward computation of one segment at a time, regenerates the needed activations, computes the gradients and then frees that memory before moving to the next segment.

The classic analysis shows that for a chain of n layers, placing checkpoints every square root of n layers reduces activation memory from order n to order square root of n, at the cost of roughly one extra forward pass. The table below summarizes where memory goes during training and which parts checkpointing affects.

Memory componentWhat determines its sizeAffected by checkpointing
Model weightsParameter count and numeric precisionNo
GradientsParameter count and precisionNo
Optimizer statesOptimizer type, for example two moments in AdamNo
ActivationsBatch size, sequence length, width, depthYes, strongly reduced
Temporary buffersKernel implementation and fragmentationSlightly
Training memory components and the effect of checkpointing

Checkpointing strategies and granularity

The main design choice is which activations to keep and which to recompute, and frameworks offer several levels of granularity. Choosing a granularity sets the balance between memory savings and recomputation overhead.

Full checkpointing of each transformer block keeps only the block input and recomputes everything inside it. Selective checkpointing keeps the expensive matrix multiplication outputs and recomputes only cheap but memory-hungry operations such as attention softmax, dropout masks and activation functions. Offloading variants move saved activations to CPU memory instead of discarding them.

StrategyWhat is storedTypical compute overheadBest suited for
Per-block full recomputationInput of each transformer blockAbout 30 to 40 percentVery large models with tight memory
Selective recomputationMost matmul outputs; recompute attention internalsAbout 2 to 10 percentLong-context transformer training
Square-root segmentationOne checkpoint every square root of n layersAbout one extra forward passUniform deep chains
Activation offloadingActivations copied to host memoryDepends on transfer bandwidthCases where recompute is costly
Partial layer checkpointingOnly some blocks checkpointedScales with share of blocksFine-tuning to just fit memory
Common gradient checkpointing strategies

In practice, selective recomputation has become popular for large language model pretraining because attention intermediates are large but cheap to recompute, so most of the memory benefit comes with a small time penalty.

Gradient checkpointing compared with other memory techniques

Gradient checkpointing is one of several complementary memory techniques, and practitioners usually combine it with others rather than choosing only one. Each method attacks a different part of the memory budget.

TechniqueMemory it reducesMain cost
Gradient checkpointingActivationsExtra forward computation
Gradient accumulationActivations per step via smaller micro-batchesMore sequential steps per update
Mixed-precision trainingActivations and weights by halving precisionNumerical stability management
Quantization with QLoRAFrozen base weightsSome accuracy and speed loss
Optimizer state shardingOptimizer states across devicesCommunication overhead
How checkpointing relates to neighboring techniques

Gradient accumulation and checkpointing are often confused. Accumulation keeps the effective batch size while processing smaller micro-batches, whereas checkpointing keeps the micro-batch size and reduces the activations stored for it. A common pattern in fine-tuning with LoRA uses both: checkpointing to fit a longer sequence and accumulation to reach the target batch size.

When to enable gradient checkpointing

Gradient checkpointing is worth enabling when activation memory is the binding constraint and the extra compute is cheaper than the alternatives. It is rarely useful when weights or optimizer states are what exceed memory.

SituationRecommended choiceReason
Out of memory at the desired sequence lengthEnable checkpointingActivations scale with sequence length
Memory fits but GPUs are underusedDisable it and raise batch sizeRecompute wastes available throughput
Weights alone exceed device memoryUse sharding or quantization firstCheckpointing does not shrink weights
Training is communication boundSelective recomputationLow overhead hides behind communication
Small model on a large GPULeave it offNo memory pressure to relieve
Decision guide for enabling checkpointing

Most frameworks expose it as a single configuration flag or a wrapper function around a module. A sound workflow is to profile peak memory first, enable checkpointing only on the layers that matter, and then use the freed memory for a larger micro-batch, which can recover part of the lost throughput.

Limitations and common mistakes

Gradient checkpointing is mathematically transparent, yet implementation details can silently change results or erase the expected savings. The following issues appear most often in practice.

  • Nondeterministic recomputation: dropout must use the same random state in the recomputed pass, otherwise the gradients no longer match the forward pass.
  • Frozen inputs without gradients: when all segment inputs are frozen, some implementations skip gradient flow, which is a frequent bug in parameter-efficient fine-tuning.
  • Incompatible caching: key-value caches used for generation must be disabled during training with checkpointing.
  • Checkpointing the wrong layers: wrapping cheap layers saves little memory while still paying recompute overhead.
  • Expecting faster training: checkpointing never speeds up a single step; any speedup comes only from the larger batch it enables.

A further misconception is that checkpointing changes model quality. With correct implementation, the computed gradients are numerically equivalent to standard backpropagation, so the learning rate, loss curve and final accuracy should not change apart from ordinary floating-point noise.

In simple terms

Training a neural network means running data forward through many layers and then working backward to learn from the mistakes. To work backward, the computer normally keeps notes about every layer, and those notes can fill up the memory of the graphics card.

Gradient checkpointing keeps only a few bookmarks instead of every note. When the backward step needs a missing note, the computer quickly redoes that part of the work from the nearest bookmark. It is like a student who saves only chapter summaries and rereads a chapter when a detail is needed. The result is the same learning with much less memory, paid for with some extra computing time.

A practical example

A research team wants to fine-tune a 7-billion-parameter language model on legal contracts with a sequence length of 8,192 tokens, using a single GPU with 80 GB of memory. With LoRA adapters and 16-bit weights, the model fits, but the first training step fails with an out-of-memory error because activations for one sequence need well over 60 GB.

The engineers enable per-block gradient checkpointing across all transformer layers. Peak memory drops to about 38 GB, which allows a micro-batch of two sequences. Each step now takes about 30 percent longer, but combined with gradient accumulation over eight steps, the team reaches its target batch size and completes the run on one GPU instead of renting a multi-GPU node.