Backpropagation is the algorithm that computes the gradient of a loss function with respect to every weight in a neural network by applying the chain rule of calculus backward from the output layer to the input layer. In neural networks, it supplies the gradients that an optimizer such as stochastic gradient descent or Adam uses to update the weights, which makes it the computational core of training nearly every modern deep learning model.
Backpropagation is often loosely used to mean the whole training procedure, but strictly it is only the gradient computation. The weight update is performed by a separate optimization algorithm, and the two together form the training loop.
How backpropagation computes gradients
Backpropagation computes gradients in two passes through the network. The forward pass feeds an input through each layer, storing intermediate values such as weighted sums and activations, and produces a prediction and a loss. The backward pass then propagates the derivative of the loss from the output toward the input, layer by layer.
At each neuron, the chain rule multiplies the incoming gradient by the local derivative of that neuron’s operation, such as the derivative of its activation function. The gradient for a weight equals the error signal at the receiving neuron multiplied by the activation of the sending neuron. Reusing these stored values is what makes the method efficient.
| Step | What happens | What is stored or produced |
|---|---|---|
| Forward pass | Inputs flow through all layers | Activations for every layer |
| Loss computation | Prediction is compared with the target | A single scalar loss value |
| Backward pass | Chain rule applied from output to input | Gradient for every weight and bias |
| Optimizer step | Weights move against the gradient | Updated parameters |
| Gradient reset | Accumulated gradients are cleared | Ready for the next mini-batch |
The key efficiency result is that the backward pass costs roughly the same order of computation as the forward pass, often quoted as about two to three times the forward cost in total. Computing each weight’s gradient separately by finite differences would require one extra forward pass per parameter, which is infeasible for models with millions or billions of weights.
Backpropagation as reverse-mode automatic differentiation
Backpropagation is a special case of reverse-mode automatic differentiation applied to a computation graph. Modern frameworks record the operations of the forward pass and then traverse the graph in reverse, so developers define only the forward computation and receive gradients automatically.
| Method | Cost for many parameters | Precision | Typical use |
|---|---|---|---|
| Reverse-mode differentiation | About one backward pass | Exact to floating point | Training neural networks |
| Forward-mode differentiation | One pass per input direction | Exact to floating point | Few inputs, many outputs |
| Finite differences | One pass per parameter | Approximate, step-size dependent | Checking gradient code |
| Symbolic differentiation | Expression growth can explode | Exact | Small analytic formulas |
The price of reverse mode is memory. All intermediate activations from the forward pass must be kept until the backward pass uses them, so activation memory grows with batch size, depth and sequence length. Gradient checkpointing trades extra computation for lower memory by recomputing selected activations during the backward pass.
Gradient flow problems and their remedies
Because backpropagation multiplies many local derivatives together, gradients can shrink toward zero or grow without bound as they travel through deep networks. These problems, called vanishing and exploding gradients, historically limited how deep networks could be trained.
| Problem | Typical cause | Common remedy |
|---|---|---|
| Vanishing gradients | Saturating activations such as sigmoid | ReLU-family activations, residual connections |
| Exploding gradients | Large weights, long recurrent chains | Gradient clipping, careful initialization |
| Dead neurons | ReLU units stuck at zero output | Leaky ReLU or GELU, lower learning rate |
| Unstable deep training | Shifting activation scales | Layer or batch normalization |
| Long sequences in RNNs | Repeated multiplication through time | Gated units such as LSTM, or a Transformer |
Weight initialization schemes such as Xavier (Glorot) and He initialization scale initial weights so that the variance of signals and gradients stays roughly constant across layers. Combined with residual connections, they allow networks with hundreds of layers to train reliably.
Backpropagation in different network architectures
The same principle applies to every differentiable architecture, but the computation graph changes its practical form. In recurrent networks, it becomes backpropagation through time, which unrolls the network over the sequence; in a Transformer, gradients flow through the attention mechanism, feed-forward blocks and residual paths.
Every component must be differentiable or approximately differentiable. Non-differentiable operations, such as discrete sampling or hard thresholds, need workarounds like the straight-through estimator or reparameterization, otherwise no useful gradient reaches the earlier layers.
In large-scale training, backpropagation runs in mixed precision, with 16-bit formats for most operations and loss scaling to prevent small gradients from underflowing. Distributed training splits the forward and backward passes across devices and averages gradients before the optimizer step.
Common misconceptions and practical pitfalls
Several recurring misunderstandings concern what backpropagation does and does not guarantee. It finds the local direction of steepest decrease of the loss; it does not find a global optimum, choose a learning rate or prevent overfitting.
- Backpropagation is not the optimizer: SGD, momentum or Adam decide the step size and direction from the gradients.
- It is not a model of how the brain learns; biological plausibility remains an open research question.
- Forgetting to reset gradients between steps causes them to accumulate and corrupts training.
- Detaching tensors accidentally stops gradient flow and silently freezes part of the model.
- A falling training loss is not success if validation loss rises, which signals overfitting.
In simple terms
Imagine a team on an assembly line that produced a faulty product. To fix it, you trace the fault backward. You ask the last worker what went wrong, then the worker before, and so on. Each one learns how much they contributed and adjusts slightly.
Backpropagation does this inside a neural network. After the network makes a wrong guess, the error is passed backward through every layer. Each connection learns how much it was responsible and is nudged in the right direction. Repeated millions of times, these small corrections are how a network learns.
A practical example
A small manufacturer trains a convolutional network to detect scratches on metal parts from camera images. Each mini-batch of 32 images runs through the network, the cross-entropy loss is computed, and backpropagation produces gradients for about eleven million weights in a single backward pass. The Adam optimizer, with a learning rate of 0.0001, then updates the weights.
Early experiments with a deeper network stall because the loss barely moves. Inspecting gradient norms per layer shows vanishing gradients in the first layers. Switching to a residual architecture with batch normalization restores healthy gradient flow, and after twenty epochs the model detects scratched parts with high recall on a held-out test set, which allows the factory to automate a manual inspection step.