An activation function is the nonlinear function a neuron applies to its weighted sum of inputs before passing the result to the next layer. In neural networks, it is the component that lets a stack of layers represent curved, interacting and conditional relationships instead of a single straight-line mapping.
Without activation functions, any number of layers would collapse into one linear transformation, so depth would add no expressive power. The choice of function also shapes how well gradients flow during training, how fast the network converges and how expensive inference becomes.
How an activation function shapes a neuron’s output
Each neuron first computes a pre-activation value z, the weighted sum of its inputs plus a bias. The activation function then maps z to an output a = f(z). This output becomes an input to the next layer, so the function decides which signals pass, how strongly and in what range.
During training, backpropagation multiplies gradients by the derivative of the activation function at every layer. If that derivative is close to zero over large input ranges, the signal fades as it travels backward, which is the vanishing gradient problem. If it is consistently above one in a deep chain, gradients can grow uncontrollably. A good activation function therefore keeps derivatives near one over the useful range of inputs.
Several properties matter when judging a function. The table below summarizes them and why each one affects training.
| Property | Why it matters | Example |
|---|---|---|
| Nonlinearity | Allows layers to model complex functions | All common choices |
| Non-saturating range | Keeps gradients from vanishing | ReLU for positive inputs |
| Zero-centered output | Balances updates across weights | Tanh, GELU roughly |
| Smoothness | Gives stable, continuous gradients | GELU, SiLU, softplus |
| Cheap computation | Lowers training and inference cost | ReLU, a single comparison |
| Bounded output | Useful for probabilities or gates | Sigmoid, tanh |
Common activation functions and their ranges
Activation functions fall into a small set of widely used families. Their output range and gradient behavior largely determine where each one fits.
| Function | Formula in brief | Output range | Typical placement |
|---|---|---|---|
| Sigmoid | 1 / (1 + e^-z) | 0 to 1 | Binary outputs, gates |
| Tanh | Scaled, shifted sigmoid | -1 to 1 | Recurrent cell states |
| ReLU | max(0, z) | 0 to infinity | Hidden layers of CNNs, MLPs |
| Leaky ReLU | z if positive, else 0.01z | Unbounded | Where dead neurons appear |
| GELU | z times Gaussian CDF of z | About -0.17 to infinity | Transformer feed-forward layers |
| SiLU or Swish | z times sigmoid of z | About -0.28 to infinity | Vision models, gated variants |
| Softmax | Exponentials normalized to sum 1 | 0 to 1, sums to 1 | Multi-class output layer |
ReLU became the default hidden-layer choice in the early 2010s because it is cheap and does not saturate for positive inputs. Modern transformer models mostly use smoother relatives such as GELU or gated forms like SwiGLU, which combine a SiLU branch with a linear branch and tend to train slightly better at large scale.
Hidden layers versus output layers
The activation function in hidden layers serves expressiveness and gradient flow, while the function in the output layer must match the task and the loss. Mixing these roles is a common source of errors.
| Task | Output activation | Paired loss |
|---|---|---|
| Binary classification | Sigmoid | Binary cross-entropy |
| Multi-class, one label | Softmax | Categorical cross-entropy |
| Multi-label classification | Sigmoid per class | Binary cross-entropy per class |
| Unbounded regression | Identity, no activation | Mean squared error |
| Positive-only regression | Softplus or exponential | Task-specific loss |
Many frameworks fuse the output activation into the loss for numerical stability, for example by accepting raw logits in a cross-entropy function. Applying softmax manually before such a loss applies it twice and distorts training.
How activation functions relate to neighboring concepts
The activation function is one part of a neuron, alongside its weights and bias. It interacts closely with backpropagation, which relies on its derivative, and with weight initialization: He initialization is designed for ReLU-family functions, while Xavier or Glorot initialization suits tanh and sigmoid.
It should not be confused with the attention mechanism, which computes weights between tokens using softmax but serves a different purpose: deciding which inputs influence each other. Normalization layers such as layer normalization also reshape signals but rescale them statistically rather than apply a fixed nonlinear curve.
Choosing an activation function and avoiding mistakes
For most new models, the architecture convention is the safest starting point: ReLU for convolutional networks and simple multilayer perceptrons, GELU or SwiGLU for transformers, and sigmoid or tanh inside recurrent gates. Changing the activation function rarely transforms results by itself, so it is best treated as a tuning step after data and architecture are sound.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Many neurons output zero permanently | Dying ReLU after large updates | Leaky ReLU, lower learning rate |
| Early layers barely learn | Saturating sigmoid or tanh | ReLU family, residual connections |
| Predicted probabilities do not sum to one | Sigmoid used for single-label task | Use softmax output |
| Regression outputs stuck in a range | Bounded output activation | Use identity output |
| Slow inference on edge hardware | Costly smooth function | ReLU or approximations |
- Do not stack two linear layers without an activation and expect extra capacity.
- Check the loss contract before adding softmax or sigmoid to the output.
- Keep initialization consistent with the chosen activation family.
A non-technical view
A neural network is made of many small units that each add up the signals they receive. The activation function is the rule that decides what each unit does with that total: stay quiet, pass it on, or pass on only part of it. It works like a dimmer switch with a shaped response, not a plain volume knob.
This matters because simple adding alone can only draw straight lines through data. Real problems, such as telling a cat from a dog, need bends and curves. The activation function adds those bends, and stacking many units with it lets the network capture very complicated patterns.
A concrete example
A machine learning engineer is training a 30-layer image classifier for sorting defects on a factory line into five categories. The first version uses sigmoid in every hidden layer, and after several epochs the early layers show gradients thousands of times smaller than the final layers. Accuracy stalls near 60 percent.
The engineer replaces the hidden sigmoids with ReLU, switches to He initialization and adds residual connections. The output layer keeps softmax with cross-entropy loss, because each part has exactly one defect class. A later check reveals that about a fifth of the neurons in one block never activate, so that block switches to Leaky ReLU with a slope of 0.01.
With these changes the model trains steadily and reaches over 90 percent accuracy on the validation set within the same compute budget, showing how the activation choice affects learning speed and stability.