Logits

On this page 7

Logits are the raw, unnormalized scores that a neural network produces in its final layer, one score per possible output class. In language models, the classes are the entries of the vocabulary, so at every generation step the model emits one logit per token, often more than 100,000 numbers, and these scores decide which token is likely to come next.

Logits sit at the boundary between the network and the decoding strategy. They are not probabilities: they can be negative, they do not sum to one, and only their relative differences carry meaning. The softmax function converts them into a probability distribution, and nearly every generation control in artificial intelligence text systems, from temperature to repetition penalty, works by adjusting logits before that conversion.

How a language model produces logits

A model produces logits by projecting its final hidden state onto the vocabulary. In a decoder-only transformer, the last layer outputs a vector of dimension d, typically 2,048 to 16,384, for each position. A matrix called the language modeling head (or unembedding matrix) multiplies that vector and yields a vector of length V, the vocabulary size.

Each entry of the result is the dot product between the hidden state and the learned vector for one token. A large positive value means the hidden state points in the same direction as that token, so the model rates it as a plausible continuation. Many models tie the output head to the input embedding matrix, which saves parameters and makes the geometry of logits mirror the geometry of token embeddings.

The table below traces the path from hidden state to sampled token in a single generation step.

StageOperationOutput
Hidden stateFinal transformer layer and normalizationVector of size d
ProjectionMultiply by the unembedding matrixLogit vector of size V
AdjustmentPenalties, bias, temperature, maskingModified logits
NormalizationSoftmax over all entriesProbability distribution
SelectionGreedy choice or samplingOne token ID
From hidden state to next token

Logits, probabilities and logprobs

The softmax function maps logits to probabilities by exponentiating each logit and dividing by the sum of all exponentials. Because of the exponent, a gap of one unit between two logits means one token is about 2.7 times as likely as the other, and a gap of five units means roughly 148 times. Adding the same constant to every logit changes nothing, which is why the absolute values are arbitrary.

Developers often confuse logits with log-probabilities, the values many APIs expose as logprobs. A logprob is the logarithm of the final probability, so it is always zero or negative and already normalized. Logits are the pre-normalization scores; subtracting the log of the softmax denominator from a logit gives its logprob.

QuantityRangeSums to oneTypical use
LogitAny real numberNoInternal scoring and decoding controls
Probability0 to 1YesSampling and calibration
LogprobNegative infinity to 0Yes, after exponentiationScoring, perplexity, confidence signals
Three representations of the same model output

The name comes from statistics, where the logit function is the log-odds of a probability. In a two-class model with a sigmoid output, the single output really is a log-odds value; in a multiclass softmax model, the word is used more loosely for any unnormalized score.

Decoding controls that operate on logits

Most text generation settings are transformations of the logit vector applied before softmax. Understanding this makes their interaction predictable: each control either rescales, shifts or masks logits, and the order in which an inference engine applies them affects the result.

ControlEffect on logitsPractical result
TemperatureDivides all logits by TBelow 1 sharpens, above 1 flattens the distribution
Top-k samplingSets all but the k highest to negative infinityRestricts choice to k candidates
Top-p samplingMasks tokens outside the smallest set reaching mass pAdaptive candidate pool
Repetition penaltyDivides or multiplies logits of seen tokensDiscourages loops
Logit biasAdds a fixed value to chosen token IDsForces or bans specific tokens
Constrained decodingMasks tokens that break a grammarGuarantees valid JSON or schema output
Common logit transformations

Greedy decoding simply takes the argmax of the logits, so temperature has no effect on it. Min-p sampling keeps tokens whose probability exceeds a fraction of the top token’s probability, which is equivalent to a threshold relative to the highest logit. Frequency and presence penalties subtract values proportional to how often a token has already appeared.

Where logits matter in practice

Logits are the main interface for inspecting and steering a model without retraining it. Teams use them in classification, evaluation, safety filtering and model compression, each relying on a different property of the scores.

Use caseHow logits are usedKey caution
Zero-shot classificationCompare logits of label tokens such as Yes and NoLabel tokenization must be checked
Knowledge distillationTrain a student to match teacher logitsNeeds access to full logits, not text
Confidence estimationMeasure the margin between top logitsRaw margins are often miscalibrated
Structured outputMask invalid tokens at each stepMasking can distort the distribution
InterpretabilityProject intermediate layers through the headEarly-layer readouts are approximate
Practical uses of logits

The logit lens technique applies the unembedding matrix to hidden states from intermediate layers, showing how the prediction forms layer by layer. Speculative decoding compares draft-model and target-model distributions derived from logits to accept or reject proposed tokens without changing the output distribution.

Limitations and common misconceptions

A high logit is not a measure of factual truth. It reflects how well a continuation fits patterns learned in training, so a model can assign a confident score to a hallucination. Instruction tuning and preference training often make output distributions sharper and less calibrated than those of the base model.

  • Logits are not comparable across models: different vocabularies, scales and training make raw values meaningless outside one model.
  • Single-token views mislead: a word split into several tokens needs the product of several probabilities, not one logit.
  • Numerical precision matters: large vocabularies in 16-bit formats risk overflow, so softmax is computed in 32-bit precision with the maximum subtracted first.
  • Access is often restricted: many hosted APIs expose only top logprobs, partly to prevent distillation of proprietary models.

Some models apply logit soft-capping, a tanh-based squashing that keeps scores within a fixed range such as plus or minus 30, which stabilizes training but changes how temperature affects sampling.

Put simply

Before a language model writes its next word, it gives a score to every word it knows. Those scores are the logits. A higher score means the word fits better, and a lower one means it fits worse. The scores are not percentages yet; a separate step turns them into odds that add up to one hundred percent.

Think of judges at a competition who write down raw points for every contestant. The points only mean something compared with each other. Settings such as temperature change those points before the winner is picked, which is how the same model can sound careful in one app and more creative in another.

In practice

A support team uses a self-hosted open-weight model to route incoming tickets into four categories: billing, technical, account and other. Instead of asking the model to write an answer and parsing free text, the engineers read the logits of the four label tokens at the position where the category should appear and apply softmax only over those four values.

This approach makes every response valid by construction and produces a usable confidence score. The team sends any ticket whose top category falls below 0.6 probability to a human agent. After calibrating the threshold on 2,000 labeled tickets, they find the automated routes match human labels in about 94 percent of cases, while roughly one ticket in eight goes to manual review. Reading logits directly replaced a fragile text parser with a single, measurable decision rule.