ReLU, short for rectified linear unit, is an activation function defined as f(x) = max(0, x): it passes positive inputs through unchanged and sets every negative input to zero. In neural networks, it is applied to the weighted sum computed by each neuron and supplies the nonlinearity that lets stacked layers represent complex functions.
ReLU became the default activation for deep networks in the early 2010s because it is cheap to compute and, unlike sigmoid and tanh, it does not saturate for positive inputs. That property keeps gradients usable through many layers and made it practical to train much deeper models.
How ReLU transforms a neuron output
A neuron first computes a pre-activation value z as the weighted sum of its inputs plus a bias, and ReLU then returns z when z is positive and 0 otherwise. The function is piecewise linear: a flat segment for negative values and an identity segment for positive values, joined at the origin.
The derivative is equally simple. It is 1 for positive inputs and 0 for negative inputs, and at exactly zero it is undefined, so frameworks conventionally use 0 there. During backpropagation, an active unit passes the gradient through unchanged, while an inactive unit blocks the gradient completely. The table summarizes these properties.
| Property | Value | Effect on training |
|---|---|---|
| Output range | 0 to positive infinity | Activations are never negative |
| Gradient for x above 0 | 1 | No shrinking of gradients |
| Gradient for x below 0 | 0 | Unit receives no update from that input |
| Cost per evaluation | One comparison | Very fast on CPUs and GPUs |
| Mean activation | Positive, not zero-centered | Slight bias shift between layers |
Although each unit is linear on either side of zero, a network of many ReLU units forms a piecewise linear function with a very large number of linear regions. This is why ReLU networks remain universal approximators despite the simplicity of the individual activation.
Why ReLU replaced sigmoid and tanh
ReLU replaced earlier activations mainly because it mitigates the vanishing gradient problem. The sigmoid derivative never exceeds 0.25, so multiplying many such factors across layers drives gradients toward zero, and tanh saturates in a similar way for large inputs.
ReLU also produces sparse activations. With typical initialization, roughly half of the units in a layer output zero for a given input, which yields representations that are efficient to store and often easier to separate. The comparison below sets ReLU against its predecessors.
| Activation | Formula | Saturation | Main drawback |
|---|---|---|---|
| Sigmoid | 1 / (1 + e^-x) | Both directions | Vanishing gradients, costly exponent |
| Tanh | (e^x – e^-x) / (e^x + e^-x) | Both directions | Vanishing gradients in deep stacks |
| ReLU | max(0, x) | Negative side only | Dead units, not zero-centered |
The practical result was a large speedup. Convolutional networks trained with ReLU have been reported to converge several times faster than equivalent tanh networks, a finding that helped establish the activation in computer vision and then across the wider field.
ReLU does not eliminate gradient problems entirely. Very deep plain stacks can still suffer from exploding or poorly scaled activations, which is why ReLU is almost always combined with careful initialization, normalization layers and residual connections in modern architectures.
Variants that address the dying ReLU problem
The best-known weakness of ReLU is the dying ReLU problem: a unit whose pre-activation becomes negative for every training example outputs zero permanently and stops learning, because its gradient is always zero. Large learning rates and strongly negative bias updates make this more likely.
Several variants keep a small signal on the negative side to prevent dead units or to produce smoother behavior. The table lists the most widely used ones.
| Variant | Negative-side behavior | Typical use |
|---|---|---|
| Leaky ReLU | Fixed slope, often 0.01 | Simple guard against dead units |
| PReLU | Learned slope per channel | Image classification networks |
| ELU | Exponential curve toward -1 | Closer to zero-centered outputs |
| GELU | Smooth, input-weighted gating | Transformer language models |
| SiLU (Swish) | Smooth, slightly negative dip | Modern vision and language models |
| ReLU6 | Zero, with output capped at 6 | Low-precision mobile inference |
Smooth variants such as GELU and SiLU now dominate many transformer architectures, often inside gated linear units. Plain ReLU nevertheless remains common in convolutional networks and in deployments where integer arithmetic and speed matter more than a small accuracy gain.
Using ReLU correctly in practice
ReLU works best when the rest of the network is configured for it. The most important companion setting is He initialization, which draws weights with a variance of 2 divided by the fan-in to compensate for the half of the signal that ReLU discards, keeping activation variance stable across layers.
The decision table below summarizes how practitioners typically choose between plain ReLU and its alternatives.
| Situation | Recommended choice | Reason |
|---|---|---|
| Standard CNN hidden layers | Plain ReLU | Fast, well understood, strong baseline |
| Many units dying in training | Leaky ReLU or lower learning rate | Restores gradient flow |
| Transformer feedforward blocks | GELU or SiLU variants | Smoother optimization at scale |
| Quantized edge deployment | ReLU or ReLU6 | Bounded, integer-friendly outputs |
| Output layer for probabilities | Softmax or sigmoid | ReLU cannot produce normalized outputs |
- Never use ReLU on the output layer of a regression task that needs negative targets.
- Monitor the fraction of zero activations per layer; values persistently near 100 percent indicate dead units.
- Pair deep ReLU stacks with normalization such as batch normalization or residual connections for stable training.
Stripped to the essentials
ReLU is a very simple rule that each artificial neuron applies to its result: if the number is positive, keep it; if it is negative, replace it with zero. That is the whole operation.
It works like a one-way valve that only lets positive signals through. Because the rule is so simple, computers can apply it billions of times quickly, and it helps learning signals travel through many layers without fading away. This small rule made very deep networks practical to train.
Where it is applied
A manufacturing company trains a convolutional network to detect scratches on metal parts from camera images on its production line. The engineers build a ResNet-style model with about 18 layers, use ReLU after every convolution and batch normalization step, and initialize weights with He initialization.
Early runs with a high learning rate leave close to 40 percent of units in one layer permanently inactive, which the team detects by logging activation statistics. Lowering the learning rate and adding a short warm-up phase brings the dead-unit share back under 5 percent.
For deployment on a small inspection computer next to the conveyor, the team converts the model to 8-bit integers. Because ReLU needs only a comparison, the quantized model processes each image in a few milliseconds, fast enough to inspect every part at full line speed.