Gradient accumulation

On this page 7

Gradient accumulation is a training technique that sums or averages gradients over several micro-batches and applies a single optimizer update only after a set number of forward and backward passes. In artificial intelligence, and especially in the training of large language models, it lets a team reach a large effective batch size on hardware whose memory can hold only a few examples at a time.

The method trades wall-clock time for memory: it does not make training faster, but it removes the hard link between the batch size an experiment needs and the batch size a single accelerator can fit. It is a standard option in nearly every modern training framework and appears in almost every fine-tuning and pretraining configuration.

How gradient accumulation works step by step

Gradient accumulation works by delaying the optimizer step while gradients from consecutive micro-batches are added into the same gradient buffers. In ordinary training, every backward pass is followed by an update and the gradients are cleared; with accumulation, the gradients are cleared only after N accumulation steps.

  1. Run the forward pass on one micro-batch and compute the loss.
  2. Divide the loss by the number of accumulation steps so the final gradient is an average, not a sum.
  3. Run the backward pass; the framework adds the new gradients to those already stored.
  4. Repeat for N micro-batches without calling the optimizer.
  5. Apply gradient clipping if configured, call the optimizer step once, and zero the gradients.

Because the gradient of a mean loss is the mean of per-example gradients, the accumulated result is mathematically equivalent to one large batch for most architectures. The key relationship is effective batch size = micro-batch size × accumulation steps × number of data-parallel devices.

The table below lists the quantities that define an accumulation setup and what each one controls.

ParameterTypical rangeWhat it controls
Micro-batch size1 to 64 sequencesPeak activation memory per device
Accumulation steps2 to 128How many passes feed one update
Data-parallel devices1 to thousandsParallel micro-batches per step
Effective batch sizeProduct of the three aboveGradient noise and update count
Optimizer steps per epochDataset size ÷ effective batchNumber of weight updates
Core parameters of a gradient accumulation setup

Why memory, not compute, motivates accumulation

Accumulation exists because activation memory grows with batch size while the model weights and optimizer states do not. During the backward pass, every layer must keep the activations of each example in the micro-batch, so doubling the batch roughly doubles that memory. Weights, gradients and Adam optimizer states stay fixed regardless of batch size.

Shrinking the micro-batch therefore cuts the largest variable cost in memory, and accumulation restores the statistical effect of the larger batch. The price is that the work is serialized rather than parallelized: the same number of examples passes through the device, but in smaller pieces that use the hardware less efficiently.

The following comparison shows how accumulation stands against related memory techniques that appear in the same Topic.

TechniqueWhat it savesMain cost
Gradient accumulationActivation memory per stepLower throughput per update
Gradient checkpointingStored activationsAbout 20-40% extra compute
Mixed-precision trainingMemory for activations and weightsNumerical range management
Optimizer state shardingOptimizer and gradient memoryExtra communication
Quantization with LoRAWeight memorySlight accuracy and speed loss
Gradient accumulation compared with other memory-saving techniques

These methods are complementary rather than competing. A typical memory-constrained fine-tuning run combines accumulation with checkpointing, mixed precision and parameter-efficient methods such as LoRA or QLoRA.

Where exact equivalence breaks down

Gradient accumulation matches a true large batch only when every operation treats examples independently. Several common components violate that assumption, and experienced practitioners check for them before trusting that two configurations behave identically.

  • Batch normalization computes statistics per micro-batch, so its normalization reflects a few examples rather than the full effective batch.
  • Variable sequence lengths make naive per-micro-batch averaging wrong: dividing each micro-batch loss by its own token count weights short sequences more heavily than a single large batch would.
  • Contrastive losses such as InfoNCE draw negatives from within the batch, so small micro-batches provide fewer negatives unless embeddings are cached across steps.
  • Dropout and random augmentation draw different random masks, which changes results slightly but does not bias them.

The loss-normalization issue is the most frequent silent bug. The correct approach for language models is to sum token losses across all micro-batches and divide once by the total number of non-padding tokens in the accumulated batch. Several popular libraries shipped incorrect normalization for years before this was widely corrected.

Choosing the number of accumulation steps

The number of accumulation steps should be chosen from the target effective batch size backward, not from what happens to fit. First decide the batch size that the learning rate and schedule were tuned for, then set the micro-batch as large as memory allows, and fill the gap with accumulation.

The decision table below summarizes typical situations and the matching strategy.

SituationRecommended strategyReason
Reproducing a published recipeMatch its effective batch exactlyLearning rate depends on batch size
Single consumer GPU fine-tuningMicro-batch 1-4, accumulate 8-32Fits memory while keeping stable updates
Large multi-node pretrainingFew or no accumulation stepsData parallelism already gives large batches
Small datasetKeep accumulation modestToo few optimizer steps per epoch
Throughput is the bottleneckRaise micro-batch before accumulationLarger kernels use the hardware better
Decision table for accumulation settings

Changing accumulation steps also changes the learning rate schedule in optimizer steps. A schedule defined in steps must be rescaled, and warmup length should be counted in updates, not in micro-batches.

Common mistakes and how to detect them

Most accumulation errors produce a run that trains but converges worse than expected, which makes them hard to notice. The table pairs typical symptoms with their causes.

SymptomLikely causeFix
Loss scaled up by NLoss not divided by accumulation stepsNormalize before the backward pass
Divergence after enabling accumulationLearning rate not adjusted for larger batchRetune or scale the learning rate
Slow multi-GPU runsGradient sync on every micro-batchSkip synchronization until the last pass
Clipping too aggressiveClipping applied per micro-batchClip once before the optimizer step
Schedule ends early or lateSteps counted in micro-batchesCount scheduler steps in updates
Symptoms, causes and fixes

In distributed training, frameworks offer a no-synchronization context that suppresses the all-reduce of gradients until the final micro-batch. Forgetting it does not change results, but it can multiply communication traffic by the number of accumulation steps.

A non-technical view

Imagine a teacher who wants to grade the opinion of a whole class of 64 students, but the room only seats eight. The teacher listens to eight students, writes down their views, invites the next eight, and keeps a running tally. Only after all 64 have spoken does the teacher make one decision. The result is the same as hearing everyone at once, it simply takes longer.

Gradient accumulation does this for a computer learning from data. The machine cannot hold many examples in memory at once, so it studies small groups, adds up what each group suggests, and adjusts itself once per large total. The benefit is that a modest computer can train like a much bigger one; the cost is time.

A concrete example

A research engineer at a university lab wants to fine-tune a 7-billion-parameter language model on a single GPU with 24 GB of memory. The original recipe used an effective batch of 64 sequences of 2,048 tokens and a learning rate of 2e-5. With QLoRA and gradient checkpointing, the GPU can fit only two sequences per micro-batch.

The engineer sets 32 accumulation steps, normalizes the loss by the total token count of each accumulated batch, applies gradient clipping once per update, and rescales the warmup to 30 optimizer steps. Each update now takes about 32 times longer than a single micro-batch, but the training curve closely tracks the reference run. The fine-tuned model reaches the recipe’s reported validation loss without access to a multi-GPU cluster.