A residual connection, also called a skip connection, is a path in a neural network that adds the input of a block directly to its output, so the block computes y = x + F(x) instead of y = F(x). In neural networks, this simple addition lets the layers learn only a correction, or residual, to their input rather than a complete new representation.
Residual connections made very deep networks trainable. They were introduced with ResNet in 2015, which trained image classifiers with more than 150 layers, and they are now a standard part of the Transformer, U-Net, diffusion models and most modern state space models. Without them, networks deeper than a few dozen layers usually train worse than shallower ones.
How a residual connection works
A residual block contains a small stack of layers, the residual branch F(x), and a parallel identity path that carries x unchanged. The two are combined by element-wise addition, which requires the input and output to have the same shape. When shapes differ, for example after downsampling, a projection shortcut such as a 1×1 convolution or a linear layer adjusts the identity path.
The key effect appears during backpropagation. The derivative of x + F(x) with respect to x is 1 plus the derivative of F, so every block passes at least an unattenuated copy of the gradient backward. Gradients from the loss reach early layers through a chain of additions instead of a long product of small factors, which directly counters the vanishing gradient problem.
The table below lists the main design parameters of a residual block and what each one affects.
| Parameter | Common choices | What it affects |
|---|---|---|
| Branch depth | 2 or 3 layers per block | Capacity of each correction |
| Shortcut type | Identity or learned projection | Shape matching and parameter count |
| Normalization position | Pre-norm or post-norm | Training stability at depth |
| Branch scaling | 1.0 or a small learned factor | Size of early updates |
| Merge operation | Addition, rarely gated addition | How information combines |
Why residual learning helps deep networks
Before residual connections, researchers observed a degradation problem: adding layers to a plain network raised the training error itself, not only the test error. This showed an optimization difficulty, not overfitting. A deeper network should be at least as good as a shallower one, because extra layers could simply learn the identity, yet plain layers find the identity hard to learn.
A residual block makes the identity the default behavior: if the branch outputs zeros, the block passes its input through unchanged. Learning a small deviation from identity is far easier than learning a full mapping, so optimizers make steady progress even with hundreds or thousands of layers.
A second interpretation views a residual network as an implicit ensemble of many shallower paths, because each block can be used or bypassed. Experiments that delete single blocks from a trained ResNet show only a gradual drop in accuracy, which supports this view.
Residual connections in major architectures
The exact form of the skip path differs between architectures. The following table compares how widely used designs implement it.
| Architecture | Form of skip path | Purpose |
|---|---|---|
| ResNet | Addition around two or three convolutions | Trainable depth in vision |
| Transformer | Addition around attention and feedforward sublayers | Stable stacking of dozens of layers |
| U-Net | Concatenation from encoder to decoder | Preserving spatial detail |
| DenseNet | Concatenation of all earlier feature maps | Feature reuse |
| Highway network | Gated addition with learned gates | Early trainable deep networks |
In a Transformer, the sequence of additions forms the residual stream, a running vector to which every attention and feedforward sublayer writes. Interpretability research often analyzes this stream directly, because it is the shared channel through which layers communicate.
Strictly speaking, the long-range links in U-Net and DenseNet are concatenation skips, not residual connections, because they stack features instead of adding them. Many texts use the term skip connection for both, so precise writing names the merge operation.
Pre-norm, post-norm and stable training
The placement of layer normalization relative to the residual addition strongly affects stability. The original Transformer used post-norm, normalizing after the addition, while most large language models now use pre-norm, normalizing inside the branch before attention or the feedforward layer, often with RMSNorm.
| Placement | Strength | Weakness |
|---|---|---|
| Post-norm | Often slightly better final quality | Needs careful learning-rate warmup |
| Pre-norm | Stable at large depth | Residual stream magnitude grows with depth |
| Sandwich or dual norm | Controls activation growth | Extra computation per block |
Initialization also matters. Techniques such as zero-initializing the last layer of each branch or scaling branch outputs by a small factor make every block start as an identity, which lets very deep models begin training without exploding activations.
Common mistakes and limitations
Residual connections are robust, but practitioners repeatedly encounter the same problems. The table below links typical symptoms to their causes and fixes.
| Symptom | Likely cause | Fix |
|---|---|---|
| Shape mismatch error | Branch changes channels or resolution | Add a projection shortcut |
| Loss diverges early | Branch outputs too large at start | Zero-init or scale the branch |
| Activation norms grow with depth | Pre-norm stream without control | Add final norm or branch scaling |
| No gain from extra depth | Activation applied after addition | Keep the identity path clean |
A clean identity path is the main design rule: nonlinearities, dropout or normalization placed directly on the shortcut weaken the gradient highway. Residual connections also add memory cost, because the input of each block must be kept for the addition, and they do not by themselves fix poor data or a mismatched architecture.
A plain-language summary
A residual connection is a shortcut that lets information skip past a group of layers. Each group only has to suggest a small change to what it receives, and that change is added on top of the original.
Think of editing a document with tracked changes instead of rewriting it from scratch at every step. The original text always survives, so even after many rounds of editing nothing important is lost. In the same way, shortcuts let very deep networks learn without losing the signal.
An example from practice
A medical imaging team trains a network to detect fractures in wrist X-rays. A plain convolutional network with 20 layers reaches good accuracy, but a 50-layer plain version trains more slowly and ends with a higher training error, a clear sign of the degradation problem.
The team switches to a ResNet-50 design with residual blocks and projection shortcuts at each downsampling stage. The deeper model now trains smoothly, converges in fewer epochs and improves sensitivity on the validation set by several percentage points. When the team later extends the model to 101 layers, training remains stable, confirming that the skip connections removed the depth barrier rather than the extra data or tuning.