Softmax, also called the softmax function, maps a vector of real-valued scores, known as logits, to a probability distribution: every output lies between 0 and 1 and all outputs sum to exactly 1. In neural networks it is the standard final step of multi-class classification and the normalization inside the attention mechanism of every Transformer.
For an input vector z with K elements, softmax computes exp(z_i) divided by the sum of exp(z_j) over all j. The exponential makes every value positive and amplifies differences between scores, so the largest logit receives a disproportionately large share of the probability mass.
How the softmax function converts logits into probabilities
Softmax works in two steps: exponentiation and normalization. Exponentiation turns any real number, including negative ones, into a positive value; dividing by the sum then rescales those values so that they form a valid distribution over K mutually exclusive classes.
The function depends only on differences between logits. Adding the same constant to every input leaves the output unchanged, a property called shift invariance. Multiplying the inputs by a factor, however, changes the result, which is why a temperature parameter has such a strong effect.
The table below shows how a three-class score vector is transformed, with values rounded to two decimals.
| Class | Logit | Exponential | Softmax probability |
|---|---|---|---|
| Cat | 2.0 | 7.39 | 0.66 |
| Dog | 1.0 | 2.72 | 0.24 |
| Bird | 0.1 | 1.11 | 0.10 |
A logit gap of one unit multiplies the odds by about 2.72, the value of e. This exponential sensitivity is why softmax outputs often look more confident than the raw score differences might suggest.
Temperature and other key parameters
The most important adjustable parameter is the temperature T, applied by dividing every logit by T before softmax. Temperature controls the sharpness of the distribution without changing the ranking of the classes.
| Temperature | Distribution shape | Typical use |
|---|---|---|
| T approaching 0 | Collapses toward one-hot (argmax) | Deterministic decoding |
| T = 1 | Standard softmax | Training and calibrated inference |
| T between 1 and 2 | Flatter, more uncertainty | Diverse text sampling |
| T well above 2 | Approaches uniform | Knowledge distillation targets |
In language models, temperature is a decoding setting: lower values produce focused, repetitive text, while higher values produce varied but less reliable output. In knowledge distillation, a high temperature exposes the relative similarities between classes that a teacher model has learned.
Softmax in classification and attention
Softmax appears in two main roles inside modern architectures. As an output layer, it converts the final logits of a classifier or language model into class or token probabilities. Inside self-attention, it converts scaled dot-product scores between queries and keys into attention weights that decide how much each token contributes.
In attention, scores are divided by the square root of the key dimension before softmax. Without this scaling, large dot products push softmax into saturated regions where gradients become vanishingly small and training slows.
Softmax is almost always paired with cross-entropy loss. The combined gradient with respect to each logit is simply the predicted probability minus the target, which is numerically clean and is one reason the pairing became the default for classification.
Softmax also appears in less visible places. Mixture of experts models use it in the gating network to weight or select experts, and contrastive learning objectives apply it across a batch of similarity scores so that each example must identify its matching partner among many negatives. In each case the function plays the same role: turning competing scores into shares of a fixed total.
How softmax compares with related functions
Softmax is one of several output functions, and the choice depends on whether classes are mutually exclusive. The comparison below covers the alternatives most often considered.
| Function | Output property | Best suited for |
|---|---|---|
| Softmax | Probabilities summing to 1 | Single-label, multi-class problems |
| Sigmoid | Independent values in 0 to 1 | Multi-label problems and binary outputs |
| Argmax | One-hot, non-differentiable | Final prediction, not training |
| Sparsemax | Probabilities with exact zeros | Sparse, interpretable attention |
| Log-softmax | Log-probabilities | Stable loss computation |
For two classes, softmax is mathematically equivalent to the sigmoid applied to the logit difference. Unlike element-wise activation functions such as ReLU, softmax couples all outputs: raising one logit lowers every other probability.
Numerical stability, cost and common mistakes
A naive implementation can overflow because exp of a logit around 89 already exceeds the float32 range. The standard fix is to subtract the maximum logit before exponentiation, which shift invariance guarantees will not change the result. Frameworks combine log-softmax and cross-entropy into one fused operation for the same reason.
Softmax also has a computational cost. In attention, it is applied over a sequence-length by sequence-length score matrix, contributing to quadratic memory growth; techniques such as linear attention and fused kernels like FlashAttention reduce or reorganize that cost. In language models with vocabularies of 50,000 to 250,000 tokens, the output softmax is a noticeable share of inference time.
| Symptom | Cause | Fix |
|---|---|---|
| NaN or infinite loss | Overflow in exp | Subtract max logit or use fused log-softmax |
| Double softmax | Softmax applied before a loss that expects logits | Pass raw logits to the loss |
| Overconfident predictions | Poor calibration | Temperature scaling on validation data |
| Wrong output for multi-label data | Softmax forces one winner | Use sigmoid per class |
A frequent misconception is that softmax probabilities are reliable confidence estimates. Deep networks are often miscalibrated, and softmax assigns high probability even to out-of-distribution inputs, so a high score does not prove the prediction is correct.
Explained without jargon
A neural network first gives each possible answer a raw score, and those scores can be any number. Softmax turns the scores into percentages that add up to 100, so the answers can be compared as chances.
It also exaggerates the leader. If one answer scores a little higher than the rest, softmax gives it a noticeably bigger share, much like a close election where the winner takes most of the seats.
A typical use case
A hospital imaging team trains a network to sort skin-lesion photos into seven diagnostic categories. The final layer outputs seven logits, and softmax turns them into probabilities such as 0.81 for one category and smaller values for the others.
During validation, the team finds the model is overconfident: cases labeled 0.9 are correct only about 75 percent of the time. They fit a single temperature of about 1.6 on held-out data, which leaves the ranking unchanged but makes the probabilities honest. Cases below a 0.6 threshold are routed to a dermatologist, so reviewer time focuses on genuinely uncertain images.