Gradient checkpointing, also called activation checkpointing, is a memory-saving technique in neural network training that stores only selected intermediate activations during the forward pass and recomputes the missing ones during the backward pass. In artificial intelligence, it is one of the standard tools for fitting large models, long sequences or larger batches into limited accelerator memory.
The technique makes a deliberate trade of compute for memory. A typical configuration adds roughly 20 to 35 percent more training time in exchange for reducing activation memory by a large factor, often enough to turn an out-of-memory failure into a working training run on the same hardware.
How gradient checkpointing reduces activation memory
Gradient checkpointing works by discarding most forward activations and keeping only a few checkpoints from which the rest can be rebuilt. Backpropagation needs the activations of every layer to compute gradients, so a standard implementation keeps all of them in memory until the backward pass consumes them.
For deep networks, these stored activations often dominate memory use, exceeding the size of the weights, gradients and optimizer states combined. Activation memory grows with batch size, sequence length, hidden width and depth, which is why long-context language models hit memory limits so quickly.
With checkpointing enabled, the framework divides the network into segments. During the forward pass it saves only the inputs to each segment. During the backward pass it reruns the forward computation of one segment at a time, regenerates the needed activations, computes the gradients and then frees that memory before moving to the next segment.
The classic analysis shows that for a chain of n layers, placing checkpoints every square root of n layers reduces activation memory from order n to order square root of n, at the cost of roughly one extra forward pass. The table below summarizes where memory goes during training and which parts checkpointing affects.
| Memory component | What determines its size | Affected by checkpointing |
|---|---|---|
| Model weights | Parameter count and numeric precision | No |
| Gradients | Parameter count and precision | No |
| Optimizer states | Optimizer type, for example two moments in Adam | No |
| Activations | Batch size, sequence length, width, depth | Yes, strongly reduced |
| Temporary buffers | Kernel implementation and fragmentation | Slightly |
Checkpointing strategies and granularity
The main design choice is which activations to keep and which to recompute, and frameworks offer several levels of granularity. Choosing a granularity sets the balance between memory savings and recomputation overhead.
Full checkpointing of each transformer block keeps only the block input and recomputes everything inside it. Selective checkpointing keeps the expensive matrix multiplication outputs and recomputes only cheap but memory-hungry operations such as attention softmax, dropout masks and activation functions. Offloading variants move saved activations to CPU memory instead of discarding them.
| Strategy | What is stored | Typical compute overhead | Best suited for |
|---|---|---|---|
| Per-block full recomputation | Input of each transformer block | About 30 to 40 percent | Very large models with tight memory |
| Selective recomputation | Most matmul outputs; recompute attention internals | About 2 to 10 percent | Long-context transformer training |
| Square-root segmentation | One checkpoint every square root of n layers | About one extra forward pass | Uniform deep chains |
| Activation offloading | Activations copied to host memory | Depends on transfer bandwidth | Cases where recompute is costly |
| Partial layer checkpointing | Only some blocks checkpointed | Scales with share of blocks | Fine-tuning to just fit memory |
In practice, selective recomputation has become popular for large language model pretraining because attention intermediates are large but cheap to recompute, so most of the memory benefit comes with a small time penalty.
Gradient checkpointing compared with other memory techniques
Gradient checkpointing is one of several complementary memory techniques, and practitioners usually combine it with others rather than choosing only one. Each method attacks a different part of the memory budget.
| Technique | Memory it reduces | Main cost |
|---|---|---|
| Gradient checkpointing | Activations | Extra forward computation |
| Gradient accumulation | Activations per step via smaller micro-batches | More sequential steps per update |
| Mixed-precision training | Activations and weights by halving precision | Numerical stability management |
| Quantization with QLoRA | Frozen base weights | Some accuracy and speed loss |
| Optimizer state sharding | Optimizer states across devices | Communication overhead |
Gradient accumulation and checkpointing are often confused. Accumulation keeps the effective batch size while processing smaller micro-batches, whereas checkpointing keeps the micro-batch size and reduces the activations stored for it. A common pattern in fine-tuning with LoRA uses both: checkpointing to fit a longer sequence and accumulation to reach the target batch size.
When to enable gradient checkpointing
Gradient checkpointing is worth enabling when activation memory is the binding constraint and the extra compute is cheaper than the alternatives. It is rarely useful when weights or optimizer states are what exceed memory.
| Situation | Recommended choice | Reason |
|---|---|---|
| Out of memory at the desired sequence length | Enable checkpointing | Activations scale with sequence length |
| Memory fits but GPUs are underused | Disable it and raise batch size | Recompute wastes available throughput |
| Weights alone exceed device memory | Use sharding or quantization first | Checkpointing does not shrink weights |
| Training is communication bound | Selective recomputation | Low overhead hides behind communication |
| Small model on a large GPU | Leave it off | No memory pressure to relieve |
Most frameworks expose it as a single configuration flag or a wrapper function around a module. A sound workflow is to profile peak memory first, enable checkpointing only on the layers that matter, and then use the freed memory for a larger micro-batch, which can recover part of the lost throughput.
Limitations and common mistakes
Gradient checkpointing is mathematically transparent, yet implementation details can silently change results or erase the expected savings. The following issues appear most often in practice.
- Nondeterministic recomputation: dropout must use the same random state in the recomputed pass, otherwise the gradients no longer match the forward pass.
- Frozen inputs without gradients: when all segment inputs are frozen, some implementations skip gradient flow, which is a frequent bug in parameter-efficient fine-tuning.
- Incompatible caching: key-value caches used for generation must be disabled during training with checkpointing.
- Checkpointing the wrong layers: wrapping cheap layers saves little memory while still paying recompute overhead.
- Expecting faster training: checkpointing never speeds up a single step; any speedup comes only from the larger batch it enables.
A further misconception is that checkpointing changes model quality. With correct implementation, the computed gradients are numerically equivalent to standard backpropagation, so the learning rate, loss curve and final accuracy should not change apart from ordinary floating-point noise.
In simple terms
Training a neural network means running data forward through many layers and then working backward to learn from the mistakes. To work backward, the computer normally keeps notes about every layer, and those notes can fill up the memory of the graphics card.
Gradient checkpointing keeps only a few bookmarks instead of every note. When the backward step needs a missing note, the computer quickly redoes that part of the work from the nearest bookmark. It is like a student who saves only chapter summaries and rereads a chapter when a detail is needed. The result is the same learning with much less memory, paid for with some extra computing time.
A practical example
A research team wants to fine-tune a 7-billion-parameter language model on legal contracts with a sequence length of 8,192 tokens, using a single GPU with 80 GB of memory. With LoRA adapters and 16-bit weights, the model fits, but the first training step fails with an out-of-memory error because activations for one sequence need well over 60 GB.
The engineers enable per-block gradient checkpointing across all transformer layers. Peak memory drops to about 38 GB, which allows a micro-batch of two sequences. Each step now takes about 30 percent longer, but combined with gradient accumulation over eight steps, the team reaches its target batch size and completes the run on one GPU instead of renting a multi-GPU node.