Cross-entropy loss, also called log loss, is the training objective that scores a model by the negative log probability it assigns to the correct answer. In artificial intelligence, it is the default loss function for classification and for the next-token prediction that drives language model training, because it rewards confident correct predictions and punishes confident mistakes very heavily.
The idea comes from information theory: cross-entropy measures how many bits or nats are needed to encode data drawn from the true distribution when the code is built from the model’s distribution. Minimizing cross-entropy therefore pushes the predicted distribution toward the empirical distribution of the training data.
How cross-entropy loss scores a prediction
Cross-entropy loss compares a predicted probability distribution with a target distribution and returns a single non-negative number. For a target distribution p and a prediction q over the same classes, the loss is the sum of p times the negative logarithm of q. With a one-hot label, every term except the correct class vanishes, so the loss reduces to the negative log of the probability given to the right class.
In practice the model outputs raw scores called logits, and a softmax function converts them into probabilities that sum to one. Frameworks fuse the softmax and the logarithm into one log-softmax operation using the log-sum-exp trick, which avoids overflow and underflow. For two classes the same idea appears as binary cross-entropy, applied to a sigmoid output.
The logarithm gives the loss its characteristic shape. The table below shows how the per-example loss changes with the probability assigned to the correct class, using the natural logarithm.
| Probability on correct class | Loss in nats | Interpretation |
|---|---|---|
| 0.99 | 0.01 | Confident and correct, almost no penalty |
| 0.90 | 0.11 | Correct with mild doubt |
| 0.50 | 0.69 | Coin flip between two options |
| 0.10 | 2.30 | Mostly wrong, strong penalty |
| 0.01 | 4.61 | Confidently wrong, severe penalty |
The batch loss is usually the mean over examples, or over tokens in a language model. Because the penalty grows without limit as the correct probability approaches zero, a few confidently wrong predictions can dominate the average, which is both a strength and a source of instability.
Why gradients of cross-entropy are well behaved
The main practical reason for choosing cross-entropy is its clean gradient with softmax outputs. The derivative of the loss with respect to each logit equals the predicted probability minus the target value. Errors therefore produce gradients proportional to the mistake, and the signal does not vanish when the model is badly wrong, unlike mean squared error applied to a saturated sigmoid.
This property makes cross-entropy pair naturally with gradient descent and adaptive optimizers such as Adam. It also explains why the loss is described as maximum likelihood estimation: minimizing average cross-entropy on labeled data is mathematically the same as maximizing the log-likelihood of the observed labels under the model.
- Gradient for the correct class is its probability minus one, so it always pushes that logit upward.
- Gradient for other classes equals their probability, so mass is removed in proportion to how much was wrongly assigned.
- Gradient magnitude shrinks as predictions approach the target, which slows learning on examples the model already handles.
Variants and related quantities
Several named variants of cross-entropy adapt the basic objective to different label formats and data problems. The choice depends on whether classes are exclusive, how balanced the data is and whether labels are fully trusted.
| Variant | Output layer | Typical use |
|---|---|---|
| Categorical cross-entropy | Softmax over classes | Single-label classification, next-token prediction |
| Binary cross-entropy | Sigmoid per output | Two-class tasks and multi-label tagging |
| Weighted cross-entropy | Softmax or sigmoid | Imbalanced classes with rare positives |
| Focal loss | Softmax or sigmoid | Dense object detection with many easy negatives |
| Label-smoothed cross-entropy | Softmax | Reducing overconfidence in large classifiers |
Cross-entropy is closely linked to other information measures. It equals the entropy of the target plus the Kullback-Leibler divergence from target to prediction. With fixed one-hot labels the entropy term is zero, so minimizing cross-entropy and minimizing KL divergence are equivalent. In language modeling, perplexity is the exponential of the mean per-token cross-entropy, so a loss of 2.0 nats corresponds to a perplexity of about 7.4.
| Quantity | Relationship to cross-entropy | Where it appears |
|---|---|---|
| Entropy | Lower bound reached when prediction equals target | Irreducible data uncertainty |
| KL divergence | Cross-entropy minus target entropy | Distillation, RLHF penalties |
| Perplexity | Exponential of mean token loss | Language model evaluation |
| Negative log-likelihood | Identical for one-hot labels | Statistical model fitting |
Cross-entropy in language model training
During pretraining, a language model predicts every next token in a sequence, and the loss is the average cross-entropy per token across the vocabulary. With vocabularies of 32,000 to more than 200,000 tokens, a uniform guess would cost the natural log of the vocabulary size, roughly 10 to 12 nats, while well-trained large models reach values near 1.5 to 2.5 nats on general web text, depending on tokenizer and data.
The same objective continues in supervised fine-tuning and instruction tuning, usually with loss masking so that only response tokens count and prompt tokens do not. In distillation, the target becomes the teacher’s full probability distribution instead of a one-hot label, which transfers richer information about which wrong answers are nearly right.
Teams monitor training curves of cross-entropy closely. A steady decline that flattens is normal, a gap between training and validation loss signals overfitting, and a sudden jump is the pattern known as a loss spike, often linked to learning rate, data quality or numerical precision problems.
Limitations and common mistakes
Cross-entropy is powerful but does not measure task quality directly. Lower loss usually tracks better accuracy, yet a model can improve its loss on easy examples while failing on the cases that matter. The table lists frequent problems and their remedies.
| Symptom | Likely cause | Fix |
|---|---|---|
| Loss becomes NaN or infinite | Log of zero from separate softmax and log | Use fused log-softmax on logits |
| Model ignores rare classes | Severe class imbalance | Class weights, resampling or focal loss |
| Overconfident wrong predictions | Loss keeps pushing probabilities to extremes | Label smoothing, calibration, early stopping |
| Noisy labels dominate training | Unbounded penalty on mislabeled data | Label cleaning or robust loss variants |
| Losses not comparable across models | Different tokenizers or vocabularies | Compare bits per byte or task metrics |
A frequent implementation error is applying softmax twice, once in the model and again inside a loss function that expects raw logits. Another is comparing loss values across tokenizers, which is misleading because the number and size of tokens differ. Finally, low cross-entropy does not guarantee well-calibrated probabilities on new data, so calibration should be checked separately.
Understanding it simply
Cross-entropy loss is a way of grading a model that does not just say an answer but says how sure it is. If the model gives the right answer a high probability, the grade is good. If it puts most of its confidence on a wrong answer, the grade is very bad.
Think of a weather forecaster who says there is a 95 percent chance of sun, and then it rains. That forecaster deserves more blame than one who said 50 percent. Cross-entropy works the same way: confident mistakes cost the most, so training teaches the model to be confident only when it has good reason.
Applied in practice
A hospital imaging team trains a classifier to sort chest X-rays into normal, pneumonia and other findings. Pneumonia appears in only about 8 percent of the 40,000 training images. With plain cross-entropy, the model reaches high accuracy by favoring the normal class, yet it misses too many pneumonia cases.
The engineers switch to weighted cross-entropy, giving the pneumonia class a weight roughly inversely proportional to its frequency, and they keep the validation loss and per-class recall under watch. They also add mild label smoothing of 0.1 because a few radiology labels are uncertain.
After retraining, overall accuracy drops slightly, but pneumonia recall rises from about 70 to 88 percent and predicted probabilities become better calibrated, which lets clinicians set a review threshold they trust. The change in loss function, not a bigger model, produced the clinically useful improvement.