ReLU

Also known as: Rectified linear unit

On this page 6

ReLU, short for rectified linear unit, is an activation function defined as f(x) = max(0, x): it passes positive inputs through unchanged and sets every negative input to zero. In neural networks, it is applied to the weighted sum computed by each neuron and supplies the nonlinearity that lets stacked layers represent complex functions.

ReLU became the default activation for deep networks in the early 2010s because it is cheap to compute and, unlike sigmoid and tanh, it does not saturate for positive inputs. That property keeps gradients usable through many layers and made it practical to train much deeper models.

How ReLU transforms a neuron output

A neuron first computes a pre-activation value z as the weighted sum of its inputs plus a bias, and ReLU then returns z when z is positive and 0 otherwise. The function is piecewise linear: a flat segment for negative values and an identity segment for positive values, joined at the origin.

The derivative is equally simple. It is 1 for positive inputs and 0 for negative inputs, and at exactly zero it is undefined, so frameworks conventionally use 0 there. During backpropagation, an active unit passes the gradient through unchanged, while an inactive unit blocks the gradient completely. The table summarizes these properties.

PropertyValueEffect on training
Output range0 to positive infinityActivations are never negative
Gradient for x above 01No shrinking of gradients
Gradient for x below 00Unit receives no update from that input
Cost per evaluationOne comparisonVery fast on CPUs and GPUs
Mean activationPositive, not zero-centeredSlight bias shift between layers
Mathematical properties of ReLU

Although each unit is linear on either side of zero, a network of many ReLU units forms a piecewise linear function with a very large number of linear regions. This is why ReLU networks remain universal approximators despite the simplicity of the individual activation.

Why ReLU replaced sigmoid and tanh

ReLU replaced earlier activations mainly because it mitigates the vanishing gradient problem. The sigmoid derivative never exceeds 0.25, so multiplying many such factors across layers drives gradients toward zero, and tanh saturates in a similar way for large inputs.

ReLU also produces sparse activations. With typical initialization, roughly half of the units in a layer output zero for a given input, which yields representations that are efficient to store and often easier to separate. The comparison below sets ReLU against its predecessors.

ActivationFormulaSaturationMain drawback
Sigmoid1 / (1 + e^-x)Both directionsVanishing gradients, costly exponent
Tanh(e^x – e^-x) / (e^x + e^-x)Both directionsVanishing gradients in deep stacks
ReLUmax(0, x)Negative side onlyDead units, not zero-centered
ReLU compared with classic activation functions

The practical result was a large speedup. Convolutional networks trained with ReLU have been reported to converge several times faster than equivalent tanh networks, a finding that helped establish the activation in computer vision and then across the wider field.

ReLU does not eliminate gradient problems entirely. Very deep plain stacks can still suffer from exploding or poorly scaled activations, which is why ReLU is almost always combined with careful initialization, normalization layers and residual connections in modern architectures.

Variants that address the dying ReLU problem

The best-known weakness of ReLU is the dying ReLU problem: a unit whose pre-activation becomes negative for every training example outputs zero permanently and stops learning, because its gradient is always zero. Large learning rates and strongly negative bias updates make this more likely.

Several variants keep a small signal on the negative side to prevent dead units or to produce smoother behavior. The table lists the most widely used ones.

VariantNegative-side behaviorTypical use
Leaky ReLUFixed slope, often 0.01Simple guard against dead units
PReLULearned slope per channelImage classification networks
ELUExponential curve toward -1Closer to zero-centered outputs
GELUSmooth, input-weighted gatingTransformer language models
SiLU (Swish)Smooth, slightly negative dipModern vision and language models
ReLU6Zero, with output capped at 6Low-precision mobile inference
Common ReLU variants

Smooth variants such as GELU and SiLU now dominate many transformer architectures, often inside gated linear units. Plain ReLU nevertheless remains common in convolutional networks and in deployments where integer arithmetic and speed matter more than a small accuracy gain.

Using ReLU correctly in practice

ReLU works best when the rest of the network is configured for it. The most important companion setting is He initialization, which draws weights with a variance of 2 divided by the fan-in to compensate for the half of the signal that ReLU discards, keeping activation variance stable across layers.

The decision table below summarizes how practitioners typically choose between plain ReLU and its alternatives.

SituationRecommended choiceReason
Standard CNN hidden layersPlain ReLUFast, well understood, strong baseline
Many units dying in trainingLeaky ReLU or lower learning rateRestores gradient flow
Transformer feedforward blocksGELU or SiLU variantsSmoother optimization at scale
Quantized edge deploymentReLU or ReLU6Bounded, integer-friendly outputs
Output layer for probabilitiesSoftmax or sigmoidReLU cannot produce normalized outputs
Choosing an activation in common situations
  • Never use ReLU on the output layer of a regression task that needs negative targets.
  • Monitor the fraction of zero activations per layer; values persistently near 100 percent indicate dead units.
  • Pair deep ReLU stacks with normalization such as batch normalization or residual connections for stable training.

Stripped to the essentials

ReLU is a very simple rule that each artificial neuron applies to its result: if the number is positive, keep it; if it is negative, replace it with zero. That is the whole operation.

It works like a one-way valve that only lets positive signals through. Because the rule is so simple, computers can apply it billions of times quickly, and it helps learning signals travel through many layers without fading away. This small rule made very deep networks practical to train.

Where it is applied

A manufacturing company trains a convolutional network to detect scratches on metal parts from camera images on its production line. The engineers build a ResNet-style model with about 18 layers, use ReLU after every convolution and batch normalization step, and initialize weights with He initialization.

Early runs with a high learning rate leave close to 40 percent of units in one layer permanently inactive, which the team detects by logging activation statistics. Lowering the learning rate and adding a short warm-up phase brings the dead-unit share back under 5 percent.

For deployment on a small inspection computer next to the conveyor, the team converts the model to 8-bit integers. Because ReLU needs only a comparison, the quantized model processes each image in a few milliseconds, fast enough to inspect every part at full line speed.