Softmax

Also known as: softmax function

On this page 7

Softmax, also called the softmax function, maps a vector of real-valued scores, known as logits, to a probability distribution: every output lies between 0 and 1 and all outputs sum to exactly 1. In neural networks it is the standard final step of multi-class classification and the normalization inside the attention mechanism of every Transformer.

For an input vector z with K elements, softmax computes exp(z_i) divided by the sum of exp(z_j) over all j. The exponential makes every value positive and amplifies differences between scores, so the largest logit receives a disproportionately large share of the probability mass.

How the softmax function converts logits into probabilities

Softmax works in two steps: exponentiation and normalization. Exponentiation turns any real number, including negative ones, into a positive value; dividing by the sum then rescales those values so that they form a valid distribution over K mutually exclusive classes.

The function depends only on differences between logits. Adding the same constant to every input leaves the output unchanged, a property called shift invariance. Multiplying the inputs by a factor, however, changes the result, which is why a temperature parameter has such a strong effect.

The table below shows how a three-class score vector is transformed, with values rounded to two decimals.

ClassLogitExponentialSoftmax probability
Cat2.07.390.66
Dog1.02.720.24
Bird0.11.110.10
Worked softmax calculation for three logits

A logit gap of one unit multiplies the odds by about 2.72, the value of e. This exponential sensitivity is why softmax outputs often look more confident than the raw score differences might suggest.

Temperature and other key parameters

The most important adjustable parameter is the temperature T, applied by dividing every logit by T before softmax. Temperature controls the sharpness of the distribution without changing the ranking of the classes.

TemperatureDistribution shapeTypical use
T approaching 0Collapses toward one-hot (argmax)Deterministic decoding
T = 1Standard softmaxTraining and calibrated inference
T between 1 and 2Flatter, more uncertaintyDiverse text sampling
T well above 2Approaches uniformKnowledge distillation targets
Effect of temperature on softmax output

In language models, temperature is a decoding setting: lower values produce focused, repetitive text, while higher values produce varied but less reliable output. In knowledge distillation, a high temperature exposes the relative similarities between classes that a teacher model has learned.

Softmax in classification and attention

Softmax appears in two main roles inside modern architectures. As an output layer, it converts the final logits of a classifier or language model into class or token probabilities. Inside self-attention, it converts scaled dot-product scores between queries and keys into attention weights that decide how much each token contributes.

In attention, scores are divided by the square root of the key dimension before softmax. Without this scaling, large dot products push softmax into saturated regions where gradients become vanishingly small and training slows.

Softmax is almost always paired with cross-entropy loss. The combined gradient with respect to each logit is simply the predicted probability minus the target, which is numerically clean and is one reason the pairing became the default for classification.

Softmax also appears in less visible places. Mixture of experts models use it in the gating network to weight or select experts, and contrastive learning objectives apply it across a batch of similarity scores so that each example must identify its matching partner among many negatives. In each case the function plays the same role: turning competing scores into shares of a fixed total.

Softmax is one of several output functions, and the choice depends on whether classes are mutually exclusive. The comparison below covers the alternatives most often considered.

FunctionOutput propertyBest suited for
SoftmaxProbabilities summing to 1Single-label, multi-class problems
SigmoidIndependent values in 0 to 1Multi-label problems and binary outputs
ArgmaxOne-hot, non-differentiableFinal prediction, not training
SparsemaxProbabilities with exact zerosSparse, interpretable attention
Log-softmaxLog-probabilitiesStable loss computation
Softmax compared with alternative output functions

For two classes, softmax is mathematically equivalent to the sigmoid applied to the logit difference. Unlike element-wise activation functions such as ReLU, softmax couples all outputs: raising one logit lowers every other probability.

Numerical stability, cost and common mistakes

A naive implementation can overflow because exp of a logit around 89 already exceeds the float32 range. The standard fix is to subtract the maximum logit before exponentiation, which shift invariance guarantees will not change the result. Frameworks combine log-softmax and cross-entropy into one fused operation for the same reason.

Softmax also has a computational cost. In attention, it is applied over a sequence-length by sequence-length score matrix, contributing to quadratic memory growth; techniques such as linear attention and fused kernels like FlashAttention reduce or reorganize that cost. In language models with vocabularies of 50,000 to 250,000 tokens, the output softmax is a noticeable share of inference time.

SymptomCauseFix
NaN or infinite lossOverflow in expSubtract max logit or use fused log-softmax
Double softmaxSoftmax applied before a loss that expects logitsPass raw logits to the loss
Overconfident predictionsPoor calibrationTemperature scaling on validation data
Wrong output for multi-label dataSoftmax forces one winnerUse sigmoid per class
Frequent softmax errors and their fixes

A frequent misconception is that softmax probabilities are reliable confidence estimates. Deep networks are often miscalibrated, and softmax assigns high probability even to out-of-distribution inputs, so a high score does not prove the prediction is correct.

Explained without jargon

A neural network first gives each possible answer a raw score, and those scores can be any number. Softmax turns the scores into percentages that add up to 100, so the answers can be compared as chances.

It also exaggerates the leader. If one answer scores a little higher than the rest, softmax gives it a noticeably bigger share, much like a close election where the winner takes most of the seats.

A typical use case

A hospital imaging team trains a network to sort skin-lesion photos into seven diagnostic categories. The final layer outputs seven logits, and softmax turns them into probabilities such as 0.81 for one category and smaller values for the others.

During validation, the team finds the model is overconfident: cases labeled 0.9 are correct only about 75 percent of the time. They fit a single temperature of about 1.6 on held-out data, which leaves the ranking unchanged but makes the probabilities honest. Cases below a 0.6 threshold are routed to a dermatologist, so reviewer time focuses on genuinely uncertain images.