A loss function, also called a cost function, is a mathematical function that measures how far a model’s predictions are from the desired targets and returns a single number that training tries to minimize. In artificial intelligence and machine learning training, it defines what good performance means for the optimizer: gradient descent adjusts the model’s parameters in the direction that lowers the loss.
Strictly, the loss usually refers to the error on one example and the cost to the average over a batch or dataset, often with regularization terms added, but the words are widely used interchangeably. The choice of loss function is one of the most consequential design decisions in training, because a model learns exactly what its loss rewards, including unintended shortcuts.
How a loss function drives training
During training, the model makes predictions for a batch, the loss function compares them with the targets, and backpropagation computes the gradient of the loss with respect to every parameter. The optimizer then moves each parameter a small step, scaled by the learning rate, against its gradient. Repeating this over many batches and epochs gradually reduces the loss.
To work with gradient-based training, a loss function must be differentiable almost everywhere and should produce informative gradients. Functions that are flat over large regions, such as raw classification accuracy, give the optimizer no direction, which is why models are trained on smooth surrogate losses and evaluated on other metrics.
The table below lists properties that make a loss function suitable for training.
| Property | Meaning | Why it matters |
|---|---|---|
| Differentiability | Gradient defined almost everywhere | Enables backpropagation |
| Alignment with the goal | Low loss means useful behavior | Prevents optimizing the wrong target |
| Numerical stability | No overflow or division by zero | Avoids failed or erratic training |
| Appropriate scale | Gradient size suits the optimizer | Keeps learning rate settings workable |
| Robustness | Limited sensitivity to outliers | Prevents a few bad labels from dominating |
Common loss functions and their uses
Each task type has standard loss functions that correspond to a probabilistic assumption about the data. Mean squared error, for example, matches the assumption of Gaussian noise, and cross-entropy matches a categorical distribution over classes or tokens.
| Loss function | Task | Key characteristic |
|---|---|---|
| Mean squared error | Regression | Penalizes large errors quadratically, sensitive to outliers |
| Mean absolute error | Regression | Linear penalty, robust but with a constant gradient |
| Huber loss | Regression with outliers | Quadratic near zero, linear for large errors |
| Cross-entropy | Classification and language modeling | Penalizes confident wrong predictions strongly |
| Binary cross-entropy | Two-class or multi-label problems | Applied per output with a sigmoid |
| Hinge loss | Margin-based classification | Used in support vector machines |
| Contrastive loss | Embedding and representation learning | Pulls matching pairs together, pushes others apart |
In large language model pretraining, the loss is next-token cross-entropy averaged over all predicted tokens. Its exponential is perplexity, so a falling training loss corresponds directly to a model that is less surprised by real text.
Loss functions compared with related concepts
The loss function is easily confused with other quantities that appear during training. Separating them clarifies what is optimized and what is merely measured.
| Concept | Role | Relation to the loss |
|---|---|---|
| Objective function | Overall quantity being optimized | Loss plus regularization, or a reward to maximize |
| Evaluation metric | Measures task success for people | Often not differentiable, so not used for gradients |
| Regularization term | Penalty on model complexity | Added to the loss to reduce overfitting |
| Reward function | Signal in reinforcement learning | Maximized rather than minimized |
| Validation loss | Same loss on held-out data | Detects overfitting and guides early stopping |
Fine-tuning methods often change the loss rather than the architecture. DPO, for instance, replaces the reward model and reinforcement learning of RLHF with a loss defined directly on pairs of preferred and rejected answers, and distillation uses a loss that matches a student model’s outputs to a teacher’s probability distribution.
Choosing and configuring a loss
The right loss follows from the output type, the noise in the labels and the cost of different errors. Practical adjustments often matter as much as the choice of the base function.
| Situation | Recommended choice | Reason |
|---|---|---|
| Heavy class imbalance | Weighted cross-entropy or focal loss | Prevents the majority class from dominating |
| Noisy regression labels | Huber or mean absolute error | Limits the influence of outliers |
| Overconfident classifier | Label smoothing, often 0.05 to 0.1 | Softens targets and improves calibration |
| Several objectives at once | Weighted sum of losses | Balances tasks, but weights need tuning |
| Ranking or retrieval | Contrastive or pairwise loss | Optimizes relative order, not absolute values |
Numerical details also matter. Frameworks compute cross-entropy directly from logits with a combined softmax and logarithm, because computing probabilities first can underflow and produce infinite loss values.
Pitfalls and misconceptions
- Lower loss is not always better behavior: a model can minimize loss by exploiting label errors or shortcuts.
- Loss values are not comparable across tasks: their scale depends on the function and the data.
- Training loss alone hides overfitting: always track validation loss alongside it.
- Mismatched loss and output layer: applying softmax twice or using the wrong reduction silently harms training.
- Exploding or NaN loss: often signals a learning rate that is too high or unstable numerics.
The deeper risk is misaligned objectives. When the loss captures only part of the real goal, optimization pushes the model toward behavior that scores well and serves users poorly, a problem closely related to reward hacking in reinforcement learning.
The simple explanation
A loss function is a score that counts mistakes. Each time a model makes a prediction, the loss function checks how wrong it was and turns that into a number. A big number means big mistakes, and a small number means the model is close.
Training is the process of making that number smaller step by step. It works like a game of hotter and colder: the score tells the model which direction leads to fewer mistakes.
How it is used in practice
A bank trains a model to flag fraudulent card transactions, where only about one transaction in a thousand is fraud. With plain cross-entropy, the model reaches a low loss by predicting not fraud almost every time, which is useless in practice.
The data science team switches to class-weighted cross-entropy and later to focal loss, which reduces the weight of easy, obvious examples and concentrates learning on the rare, difficult ones. They monitor recall and precision on a validation set rather than accuracy. The new model catches substantially more fraud at the same alert volume, and the team documents the chosen loss and weights so that later retraining keeps the same priorities.