Cross-entropy loss

Also known as: log loss

On this page 7

Cross-entropy loss, also called log loss, is the training objective that scores a model by the negative log probability it assigns to the correct answer. In artificial intelligence, it is the default loss function for classification and for the next-token prediction that drives language model training, because it rewards confident correct predictions and punishes confident mistakes very heavily.

The idea comes from information theory: cross-entropy measures how many bits or nats are needed to encode data drawn from the true distribution when the code is built from the model’s distribution. Minimizing cross-entropy therefore pushes the predicted distribution toward the empirical distribution of the training data.

How cross-entropy loss scores a prediction

Cross-entropy loss compares a predicted probability distribution with a target distribution and returns a single non-negative number. For a target distribution p and a prediction q over the same classes, the loss is the sum of p times the negative logarithm of q. With a one-hot label, every term except the correct class vanishes, so the loss reduces to the negative log of the probability given to the right class.

In practice the model outputs raw scores called logits, and a softmax function converts them into probabilities that sum to one. Frameworks fuse the softmax and the logarithm into one log-softmax operation using the log-sum-exp trick, which avoids overflow and underflow. For two classes the same idea appears as binary cross-entropy, applied to a sigmoid output.

The logarithm gives the loss its characteristic shape. The table below shows how the per-example loss changes with the probability assigned to the correct class, using the natural logarithm.

Probability on correct classLoss in natsInterpretation
0.990.01Confident and correct, almost no penalty
0.900.11Correct with mild doubt
0.500.69Coin flip between two options
0.102.30Mostly wrong, strong penalty
0.014.61Confidently wrong, severe penalty
Per-example loss for different probabilities on the correct class

The batch loss is usually the mean over examples, or over tokens in a language model. Because the penalty grows without limit as the correct probability approaches zero, a few confidently wrong predictions can dominate the average, which is both a strength and a source of instability.

Why gradients of cross-entropy are well behaved

The main practical reason for choosing cross-entropy is its clean gradient with softmax outputs. The derivative of the loss with respect to each logit equals the predicted probability minus the target value. Errors therefore produce gradients proportional to the mistake, and the signal does not vanish when the model is badly wrong, unlike mean squared error applied to a saturated sigmoid.

This property makes cross-entropy pair naturally with gradient descent and adaptive optimizers such as Adam. It also explains why the loss is described as maximum likelihood estimation: minimizing average cross-entropy on labeled data is mathematically the same as maximizing the log-likelihood of the observed labels under the model.

  • Gradient for the correct class is its probability minus one, so it always pushes that logit upward.
  • Gradient for other classes equals their probability, so mass is removed in proportion to how much was wrongly assigned.
  • Gradient magnitude shrinks as predictions approach the target, which slows learning on examples the model already handles.

Several named variants of cross-entropy adapt the basic objective to different label formats and data problems. The choice depends on whether classes are exclusive, how balanced the data is and whether labels are fully trusted.

VariantOutput layerTypical use
Categorical cross-entropySoftmax over classesSingle-label classification, next-token prediction
Binary cross-entropySigmoid per outputTwo-class tasks and multi-label tagging
Weighted cross-entropySoftmax or sigmoidImbalanced classes with rare positives
Focal lossSoftmax or sigmoidDense object detection with many easy negatives
Label-smoothed cross-entropySoftmaxReducing overconfidence in large classifiers
Common variants and when they apply

Cross-entropy is closely linked to other information measures. It equals the entropy of the target plus the Kullback-Leibler divergence from target to prediction. With fixed one-hot labels the entropy term is zero, so minimizing cross-entropy and minimizing KL divergence are equivalent. In language modeling, perplexity is the exponential of the mean per-token cross-entropy, so a loss of 2.0 nats corresponds to a perplexity of about 7.4.

QuantityRelationship to cross-entropyWhere it appears
EntropyLower bound reached when prediction equals targetIrreducible data uncertainty
KL divergenceCross-entropy minus target entropyDistillation, RLHF penalties
PerplexityExponential of mean token lossLanguage model evaluation
Negative log-likelihoodIdentical for one-hot labelsStatistical model fitting
How cross-entropy relates to neighboring measures

Cross-entropy in language model training

During pretraining, a language model predicts every next token in a sequence, and the loss is the average cross-entropy per token across the vocabulary. With vocabularies of 32,000 to more than 200,000 tokens, a uniform guess would cost the natural log of the vocabulary size, roughly 10 to 12 nats, while well-trained large models reach values near 1.5 to 2.5 nats on general web text, depending on tokenizer and data.

The same objective continues in supervised fine-tuning and instruction tuning, usually with loss masking so that only response tokens count and prompt tokens do not. In distillation, the target becomes the teacher’s full probability distribution instead of a one-hot label, which transfers richer information about which wrong answers are nearly right.

Teams monitor training curves of cross-entropy closely. A steady decline that flattens is normal, a gap between training and validation loss signals overfitting, and a sudden jump is the pattern known as a loss spike, often linked to learning rate, data quality or numerical precision problems.

Limitations and common mistakes

Cross-entropy is powerful but does not measure task quality directly. Lower loss usually tracks better accuracy, yet a model can improve its loss on easy examples while failing on the cases that matter. The table lists frequent problems and their remedies.

SymptomLikely causeFix
Loss becomes NaN or infiniteLog of zero from separate softmax and logUse fused log-softmax on logits
Model ignores rare classesSevere class imbalanceClass weights, resampling or focal loss
Overconfident wrong predictionsLoss keeps pushing probabilities to extremesLabel smoothing, calibration, early stopping
Noisy labels dominate trainingUnbounded penalty on mislabeled dataLabel cleaning or robust loss variants
Losses not comparable across modelsDifferent tokenizers or vocabulariesCompare bits per byte or task metrics
Typical problems with cross-entropy and how to address them

A frequent implementation error is applying softmax twice, once in the model and again inside a loss function that expects raw logits. Another is comparing loss values across tokenizers, which is misleading because the number and size of tokens differ. Finally, low cross-entropy does not guarantee well-calibrated probabilities on new data, so calibration should be checked separately.

Understanding it simply

Cross-entropy loss is a way of grading a model that does not just say an answer but says how sure it is. If the model gives the right answer a high probability, the grade is good. If it puts most of its confidence on a wrong answer, the grade is very bad.

Think of a weather forecaster who says there is a 95 percent chance of sun, and then it rains. That forecaster deserves more blame than one who said 50 percent. Cross-entropy works the same way: confident mistakes cost the most, so training teaches the model to be confident only when it has good reason.

Applied in practice

A hospital imaging team trains a classifier to sort chest X-rays into normal, pneumonia and other findings. Pneumonia appears in only about 8 percent of the 40,000 training images. With plain cross-entropy, the model reaches high accuracy by favoring the normal class, yet it misses too many pneumonia cases.

The engineers switch to weighted cross-entropy, giving the pneumonia class a weight roughly inversely proportional to its frequency, and they keep the validation loss and per-class recall under watch. They also add mild label smoothing of 0.1 because a few radiology labels are uncertain.

After retraining, overall accuracy drops slightly, but pneumonia recall rises from about 70 to 88 percent and predicted probabilities become better calibrated, which lets clinicians set a review threshold they trust. The change in loss function, not a bigger model, produced the clinically useful improvement.