Residual connection

Also known as: skip connection

On this page 7

A residual connection, also called a skip connection, is a path in a neural network that adds the input of a block directly to its output, so the block computes y = x + F(x) instead of y = F(x). In neural networks, this simple addition lets the layers learn only a correction, or residual, to their input rather than a complete new representation.

Residual connections made very deep networks trainable. They were introduced with ResNet in 2015, which trained image classifiers with more than 150 layers, and they are now a standard part of the Transformer, U-Net, diffusion models and most modern state space models. Without them, networks deeper than a few dozen layers usually train worse than shallower ones.

How a residual connection works

A residual block contains a small stack of layers, the residual branch F(x), and a parallel identity path that carries x unchanged. The two are combined by element-wise addition, which requires the input and output to have the same shape. When shapes differ, for example after downsampling, a projection shortcut such as a 1×1 convolution or a linear layer adjusts the identity path.

The key effect appears during backpropagation. The derivative of x + F(x) with respect to x is 1 plus the derivative of F, so every block passes at least an unattenuated copy of the gradient backward. Gradients from the loss reach early layers through a chain of additions instead of a long product of small factors, which directly counters the vanishing gradient problem.

The table below lists the main design parameters of a residual block and what each one affects.

ParameterCommon choicesWhat it affects
Branch depth2 or 3 layers per blockCapacity of each correction
Shortcut typeIdentity or learned projectionShape matching and parameter count
Normalization positionPre-norm or post-normTraining stability at depth
Branch scaling1.0 or a small learned factorSize of early updates
Merge operationAddition, rarely gated additionHow information combines
Design parameters of a residual block

Why residual learning helps deep networks

Before residual connections, researchers observed a degradation problem: adding layers to a plain network raised the training error itself, not only the test error. This showed an optimization difficulty, not overfitting. A deeper network should be at least as good as a shallower one, because extra layers could simply learn the identity, yet plain layers find the identity hard to learn.

A residual block makes the identity the default behavior: if the branch outputs zeros, the block passes its input through unchanged. Learning a small deviation from identity is far easier than learning a full mapping, so optimizers make steady progress even with hundreds or thousands of layers.

A second interpretation views a residual network as an implicit ensemble of many shallower paths, because each block can be used or bypassed. Experiments that delete single blocks from a trained ResNet show only a gradual drop in accuracy, which supports this view.

Residual connections in major architectures

The exact form of the skip path differs between architectures. The following table compares how widely used designs implement it.

ArchitectureForm of skip pathPurpose
ResNetAddition around two or three convolutionsTrainable depth in vision
TransformerAddition around attention and feedforward sublayersStable stacking of dozens of layers
U-NetConcatenation from encoder to decoderPreserving spatial detail
DenseNetConcatenation of all earlier feature mapsFeature reuse
Highway networkGated addition with learned gatesEarly trainable deep networks
Skip paths across common architectures

In a Transformer, the sequence of additions forms the residual stream, a running vector to which every attention and feedforward sublayer writes. Interpretability research often analyzes this stream directly, because it is the shared channel through which layers communicate.

Strictly speaking, the long-range links in U-Net and DenseNet are concatenation skips, not residual connections, because they stack features instead of adding them. Many texts use the term skip connection for both, so precise writing names the merge operation.

Pre-norm, post-norm and stable training

The placement of layer normalization relative to the residual addition strongly affects stability. The original Transformer used post-norm, normalizing after the addition, while most large language models now use pre-norm, normalizing inside the branch before attention or the feedforward layer, often with RMSNorm.

PlacementStrengthWeakness
Post-normOften slightly better final qualityNeeds careful learning-rate warmup
Pre-normStable at large depthResidual stream magnitude grows with depth
Sandwich or dual normControls activation growthExtra computation per block
Comparison of normalization placements

Initialization also matters. Techniques such as zero-initializing the last layer of each branch or scaling branch outputs by a small factor make every block start as an identity, which lets very deep models begin training without exploding activations.

Common mistakes and limitations

Residual connections are robust, but practitioners repeatedly encounter the same problems. The table below links typical symptoms to their causes and fixes.

SymptomLikely causeFix
Shape mismatch errorBranch changes channels or resolutionAdd a projection shortcut
Loss diverges earlyBranch outputs too large at startZero-init or scale the branch
Activation norms grow with depthPre-norm stream without controlAdd final norm or branch scaling
No gain from extra depthActivation applied after additionKeep the identity path clean
Troubleshooting residual blocks

A clean identity path is the main design rule: nonlinearities, dropout or normalization placed directly on the shortcut weaken the gradient highway. Residual connections also add memory cost, because the input of each block must be kept for the addition, and they do not by themselves fix poor data or a mismatched architecture.

A plain-language summary

A residual connection is a shortcut that lets information skip past a group of layers. Each group only has to suggest a small change to what it receives, and that change is added on top of the original.

Think of editing a document with tracked changes instead of rewriting it from scratch at every step. The original text always survives, so even after many rounds of editing nothing important is lost. In the same way, shortcuts let very deep networks learn without losing the signal.

An example from practice

A medical imaging team trains a network to detect fractures in wrist X-rays. A plain convolutional network with 20 layers reaches good accuracy, but a 50-layer plain version trains more slowly and ends with a higher training error, a clear sign of the degradation problem.

The team switches to a ResNet-50 design with residual blocks and projection shortcuts at each downsampling stage. The deeper model now trains smoothly, converges in fewer epochs and improves sensitivity on the validation set by several percentage points. When the team later extends the model to 101 layers, training remains stable, confirming that the skip connections removed the depth barrier rather than the extra data or tuning.