Gradient clipping

On this page 8

Gradient clipping is a training technique that limits the magnitude of gradients before the optimizer applies them, so that an unusually large gradient cannot push model weights into an unstable region. Training deep networks in artificial intelligence, especially recurrent networks and large transformer language models, routinely relies on it as a cheap safeguard against exploding gradients and sudden loss spikes.

The method does not change which direction the model learns in under the most common variant; it only rescales oversized updates. That makes it a stability tool rather than an optimization method in its own right, and it works alongside the learning rate, the optimizer and the learning rate schedule rather than replacing any of them.

How gradient clipping works

Gradient clipping works by measuring the gradient after backpropagation and shrinking it whenever it exceeds a chosen threshold. The check happens once per optimizer step, after the loss has been differentiated and, in distributed training, after gradients have been synchronized across devices.

In the dominant variant, clipping by global norm, the framework computes the L2 norm of all parameter gradients concatenated into one vector. If that norm exceeds the maximum norm threshold, every gradient is multiplied by the ratio of the threshold to the measured norm. The update keeps its direction unchanged but its length is capped.

The steps below show the order in which a typical training loop applies clipping:

  1. Run the forward pass and compute the loss for the batch.
  2. Backpropagate to obtain gradients for every trainable parameter.
  3. If mixed-precision training is used, unscale the gradients first.
  4. Compute the global gradient norm and compare it with the threshold.
  5. Rescale all gradients by threshold divided by norm when the norm is larger.
  6. Let the optimizer, for example the Adam optimizer, apply the update.

Variants of gradient clipping

Several variants of gradient clipping exist, and they differ in what quantity is bounded and whether direction is preserved. The table compares the forms that practitioners encounter most often.

VariantWhat is limitedEffect on directionTypical use
Global norm clippingL2 norm of all gradients togetherPreservedTransformers, large language models
Per-parameter norm clippingNorm of each tensor separatelyChanged across tensorsOlder recurrent network code
Value clippingEach gradient element to a fixed rangeChangedSimple models, reinforcement learning
Adaptive gradient clippingGradient norm relative to weight normPreserved per unitNormalizer-free image networks
Per-example clippingGradient of each training exampleChanged for the batchDifferentially private training
Main forms of gradient clipping

Value clipping is the bluntest form: an element larger than the bound is set to the bound, which distorts the update direction when only a few coordinates are large. Adaptive gradient clipping scales the limit with the size of the weights it updates, so small layers are not throttled by a single global number. Per-example clipping is required by differentially private stochastic gradient descent, where it bounds each example’s influence for the privacy guarantee rather than for stability.

Key parameters and typical values

The behavior of gradient clipping is governed by a small set of hyperparameters and implementation choices. Typical values depend on model size, optimizer and batch size, so the ranges below are starting points rather than rules.

SettingCommon rangeWhat it affects
Maximum global norm0.5 to 5.0, often 1.0How often and how hard updates are capped
Value clip bound0.1 to 10 per elementLargest allowed single coordinate
Norm typeL2, rarely L-infinityHow the gradient size is measured
Clipping frequencyEvery optimizer stepConsistency of the safeguard
Position with accumulationAfter the last micro-batchWhether the full batch is measured
Settings that control clipping behavior

A maximum norm of 1.0 has become a common default for transformer pretraining with AdamW. The right value is best found by logging the gradient norm during early training: a threshold slightly above the typical norm clips only outliers, while a threshold below it clips almost every step and silently acts as a reduced learning rate.

Gradient clipping is one of several tools that keep optimization stable, and it is most effective when combined with them. It treats the symptom of oversized updates, while other techniques address the causes or the overall step size.

TechniqueMain purposeRelation to clipping
Learning rateScales every updateClipping bounds only unusual steps
Learning rate warmupAvoids early instabilityReduces how often clipping triggers
Weight decayRegularizes weight sizeIndependent; applied in the optimizer
Normalization layersKeep activations well scaledPrevent many explosions at the source
Loss scalingAvoids fp16 underflowGradients must be unscaled before clipping
How clipping relates to neighboring techniques

A frequent confusion concerns gradient accumulation. When several micro-batches are accumulated into one effective batch, clipping must happen once, after the final micro-batch; clipping each micro-batch separately changes the effective threshold and the meaning of the batch size.

When to apply gradient clipping

Gradient clipping is worth enabling whenever a model has deep or recurrent computation paths, a high learning rate, or noisy data that can produce rare extreme batches. The decision table summarizes common situations.

SituationRecommended choiceReason
Pretraining a transformerGlobal norm at about 1.0Prevents rare loss spikes from derailing the run
Training a recurrent network on long sequencesGlobal norm between 1 and 5Products of Jacobians grow over time steps
Fine-tuning with LoRA or full weightsKeep the base recipe, often 1.0Small datasets give noisy batches
Policy gradient reinforcement learningGlobal norm about 0.5Reward variance creates large updates
Small, well-normalized vision modelOptional or noneGradients rarely explode
Choosing a clipping strategy

Limitations and common mistakes

Gradient clipping is not a cure for a broken setup. If the norm exceeds the threshold on most steps, the real problem is usually a learning rate that is too high, bad initialization, corrupted data or a numerical bug, and clipping only hides it.

  • Clipping before unscaling in mixed-precision training compares the scaled gradient with the threshold and makes clipping nearly constant.
  • Threshold set too low turns clipping into a hidden learning rate reduction and slows convergence.
  • Not logging the pre-clip norm removes the earliest warning signal of instability.
  • Clipping per device before synchronization in data-parallel training gives each replica a different update.
  • Expecting clipping to fix vanishing gradients is a misconception; it only limits large gradients.

With adaptive optimizers such as Adam, the per-parameter normalization already damps many large gradients, yet a sudden spike can still corrupt the moment estimates for many steps. Clipping protects those running statistics, which is why it remains standard even with adaptive optimizers.

Explained without jargon

A model learns by making many small corrections, one batch of examples at a time. Occasionally a strange batch suggests an enormous correction, and following it blindly can undo hours of progress.

Gradient clipping works like a speed limiter on a car. The driver still chooses the direction, but the car never goes faster than a set limit. The model keeps learning in the direction the data suggests, yet no single step can be too big, so training stays steady.

A typical use case

A research team pretrains a 1.3-billion-parameter language model on 64 GPUs with AdamW, a peak learning rate of 3e-4 and bf16 mixed precision. Around step 18,000 the logged loss jumps sharply after a batch containing long, repetitive machine-generated text, and the pre-clip gradient norm rises from its usual 0.3 to over 40.

Because the training configuration applies global norm clipping at 1.0, that update is scaled down by a factor of about 40 before it reaches the optimizer. The loss returns to its trend within roughly 200 steps instead of diverging, and the team avoids restarting from an earlier checkpoint. They also add the offending data source to a filter list, treating the logged spike as a data quality signal rather than relying on clipping alone.