Vanishing gradient problem

Also known as: vanishing gradients

On this page 8

The vanishing gradient problem is the tendency of error gradients to shrink exponentially as backpropagation carries them from the output layer toward the input layer of deep neural networks. When gradients become tiny, the weights of early layers barely change, and training stalls or becomes extremely slow even though the loss has not converged.

The problem, often described simply as vanishing gradients, was identified in the early 1990s in the analysis of recurrent networks and long held back deep learning. Most standard components of modern architectures, from ReLU activations to residual connections and normalization layers, exist partly to keep gradients at a healthy magnitude across many layers.

Why gradients vanish during backpropagation

Gradients vanish because backpropagation applies the chain rule, which multiplies one local derivative per layer. The gradient that reaches layer one is a product of terms from every later layer: activation derivatives and weight matrices. If these factors are consistently smaller than one in magnitude, the product decays exponentially with depth.

The classic culprit is the sigmoid activation, whose derivative never exceeds 0.25. Across ten sigmoid layers, the activation factor alone can shrink a gradient by a factor of roughly 0.25 to the tenth power, about one millionth. The hyperbolic tangent is better, with a maximum derivative of 1, but it also saturates: for large positive or negative inputs its slope approaches zero.

The table below lists the maximum derivative of common activation functions and what that means for gradient flow.

ActivationMaximum derivativeSaturation behaviorGradient risk
Sigmoid0.25Saturates at both endsHigh
Tanh1.0Saturates at both endsModerate to high
ReLU1.0Zero for negative inputsLow, but dead units possible
Leaky ReLU1.0Small slope for negative inputsLow
GELU and SiLUAbout 1.1Smooth, near-linear for positive inputsLow
Activation functions and their effect on gradient flow

Vanishing gradients in recurrent networks

Recurrent neural networks suffer especially badly because backpropagation through time multiplies the same recurrent weight matrix once per time step. A sequence of 100 steps behaves like a 100-layer network with shared weights, so the influence of early inputs on the loss fades almost completely.

The practical consequence is that a plain RNN cannot learn long-range dependencies, such as agreement between a subject and a verb separated by many words. When the largest singular value of the recurrent matrix is below one, gradients vanish; when it exceeds one, they tend to explode instead. The LSTM was designed specifically to address this with a gated memory cell whose additive update lets gradients pass across many steps nearly unchanged.

Symptoms and how to diagnose the problem

Vanishing gradients are diagnosed by monitoring gradient norms per layer during training. In a healthy network, norms differ between layers by perhaps one or two orders of magnitude; in an affected network, early-layer norms can be many orders smaller than late-layer norms.

The table below connects observable symptoms with their likely cause and a first corrective step.

SymptomLikely causeFirst fix
Loss plateaus early at a high valueEarly layers receive near-zero gradientsSwitch to ReLU-family activations
Early-layer weights barely changeSaturated sigmoid or tanh unitsApply proper weight initialization
Deeper model trains worse than shallower oneSignal degrades through depthAdd residual connections
RNN ignores distant contextRepeated recurrent multiplicationUse LSTM or GRU cells
Many ReLU units output zero permanentlyDying ReLU after large updatesUse Leaky ReLU or lower learning rate
Symptoms of vanishing gradients and typical fixes

A useful check is to plot the ratio of gradient norm to weight norm for each layer. Ratios far below about 0.0001 in early layers suggest that those layers are effectively frozen.

Techniques that counter vanishing gradients

No single fix solves the vanishing gradient problem; modern architectures combine several complementary techniques. Each one attacks a different factor in the chain-rule product, either the activation derivative, the weight scale or the path length.

TechniqueMechanismTypical use
ReLU-family activationsDerivative of 1 for active unitsAlmost all feedforward and convolutional networks
Careful weight initializationKeeps signal variance stable across layersXavier for tanh, He for ReLU
Residual connectionsIdentity path lets gradients bypass layersResNet, transformers
Normalization layersKeeps activations in a non-saturated rangeBatch normalization, layer normalization, RMSNorm
Gated recurrent cellsAdditive memory updates preserve gradientsLSTM, GRU
Layer-wise pretrainingInitializes layers one at a timeHistorical, before 2012
Main remedies and the mechanism behind each

Residual connections deserve special attention. Because a residual block computes the input plus a learned correction, the derivative of each block contains an identity term, so gradients always have a direct route back to early layers. This is why networks with more than 100 layers became trainable after ResNet appeared in 2015.

Weight initialization sets the starting scale. He initialization draws weights with variance of two divided by the number of inputs, which compensates for ReLU zeroing roughly half of the activations and keeps signal variance roughly constant through depth.

Vanishing compared with exploding gradients

The vanishing gradient problem and the exploding gradient problem are two sides of the same instability: both arise from multiplying many factors during backpropagation. The difference is whether those factors are mostly below or above one in magnitude.

AspectVanishing gradientsExploding gradients
Factor magnitudeMostly below 1Mostly above 1
Visible effectTraining stalls silentlyLoss spikes or becomes NaN
Ease of detectionHard, failure is quietEasy, failure is loud
Main remediesReLU, residuals, gated cellsGradient clipping, lower learning rate
Vanishing and exploding gradients side by side

The asymmetry in detection matters. Exploding gradients announce themselves with numerical errors, while vanishing gradients fail quietly, so a model can appear to train normally while its early layers learn almost nothing.

Common misconceptions about vanishing gradients

  • It is not fully solved: very deep or very long-sequence models can still show weak gradient flow, particularly when residual paths or normalization are misconfigured.
  • ReLU alone is not enough: without suitable initialization and normalization, deep ReLU networks can still degrade or develop many dead units.
  • Small gradients are not always a problem: near convergence, gradients should shrink; the issue is a large imbalance between layers early in training.
  • A larger learning rate does not fix it: raising the step size scales every layer equally and usually destabilizes the later layers first.

Put simply

A neural network learns by passing a correction message backward from its output to its first layers. In a deep network, that message travels through many layers, and each layer can make it a little weaker. The vanishing gradient problem is what happens when the message fades almost to nothing before it reaches the beginning.

It resembles a long game of whispered messages in a large room where each person speaks a bit more quietly than the last. The people at the far end hear nothing useful, so they never adjust. Engineers fix this with shortcuts that carry the message directly, better starting settings and building blocks that do not muffle the signal.

In practice

A machine learning engineer builds a 20-layer fully connected network with sigmoid activations to predict equipment failures from sensor data. After several hours of training, the loss stays nearly flat. Logging gradient norms per layer shows that the last layer receives gradients around 0.01, while the first layer receives values near one ten-millionth, so the early layers are effectively frozen.

The engineer replaces sigmoid with ReLU, switches to He initialization and adds residual connections every two layers together with layer normalization. On the next run, gradient norms across layers stay within about two orders of magnitude, the loss falls steadily within the first epoch, and validation accuracy improves markedly. The fix required no additional data, only changes that restored gradient flow.