Label smoothing

On this page 8

Label smoothing is a regularization technique used in the training of classifiers and language models that replaces hard one-hot targets with a softened target distribution, assigning most probability to the correct class and spreading a small amount across the others. In artificial intelligence, it is applied to the cross-entropy loss so that a network is no longer rewarded for pushing the logit of the correct class toward infinity.

The technique was popularized in image classification and quickly became a default in neural machine translation and transformer training. Its effect is subtle but well documented: it usually improves calibration and generalization slightly, while changing how the network organizes its internal representations in ways that matter for later uses such as distillation.

How label smoothing modifies the training target

Label smoothing changes the target vector, not the model. For a problem with K classes and a smoothing parameter epsilon, the correct class receives a target probability of 1 minus epsilon plus epsilon divided by K, and every other class receives epsilon divided by K. With epsilon set to 0.1 and ten classes, the correct class target becomes 0.91 and each wrong class receives 0.01.

The loss is still cross-entropy, but the minimum is now reached at a finite logit gap rather than an unbounded one. Mathematically, the smoothed loss equals the ordinary cross-entropy plus a term proportional to the cross-entropy between the uniform distribution and the model output, which penalizes predictions that drive any class probability close to zero. That penalty is what curbs overconfident predictions.

The following table shows how the target for the correct class changes with typical settings.

EpsilonClassesCorrect class targetEach other class
0.0101.0000.000
0.1100.9100.010
0.11,0000.90010.0001
0.2100.8200.020
0.150,000 tokensAbout 0.900About 0.000002
Target values under different smoothing settings

Because the smoothing mass is divided by K, large vocabularies spread it extremely thin. In a language model with tens of thousands of tokens, almost all of the effect comes from lowering the correct target rather than from raising specific alternatives.

Variants and typical parameter values

Several forms of label smoothing exist, and they differ in how the smoothing mass is distributed. The uniform version is by far the most common, but some tasks benefit from a prior that reflects real class similarity.

VariantHow the mass is spreadTypical use
Uniform smoothingEqually across all classesImage classification, translation
Unigram prior smoothingIn proportion to class frequencyLanguage modeling, speech recognition
Online label smoothingFrom the model’s own past predictionsFine-grained classification
Confidence penaltyPenalizes low output entropy directlyAlternative regularizer
Teacher soft labelsFrom a trained teacher modelDistillation
Common forms of label smoothing

In practice, epsilon between 0.05 and 0.2 covers most published configurations, and 0.1 is the conventional default. Values above about 0.3 usually hurt accuracy because the target no longer distinguishes the correct class sharply enough. The setting is a hyperparameter that interacts with the learning rate, weight decay and the number of classes, so it should be tuned rather than copied blindly.

Label smoothing belongs to a family of methods that soften the supervision signal, but each method changes a different part of training. Confusing them leads to poor choices, especially when distillation is planned later.

MethodWhat changesInformation in the targetMain goal
Label smoothingTarget distributionFixed and class-agnosticCalibration, regularization
Knowledge distillationTarget from a teacherRich class similarityCompress a model
MixupInputs and targets togetherInterpolated pairsSmoother decision boundaries
Weight decayParameter magnitudeNoneLimit model complexity
DropoutNetwork activationsNoneReduce co-adaptation
Label smoothing and neighboring methods

The key difference from distillation is that uniform smoothing carries no knowledge about which wrong classes are plausible. A teacher that says a husky looks somewhat like a wolf transmits structure; label smoothing simply says every wrong class is equally slightly possible.

Effects on calibration and learned representations

The best-established benefit of label smoothing is better calibration, meaning that predicted confidence tracks real accuracy more closely. Networks trained on hard targets often report 99 percent confidence on inputs they classify wrongly, while smoothed networks typically show a lower expected calibration error without a separate temperature scaling step.

Label smoothing also reshapes the penultimate-layer representations. Experiments show that examples of each class form tight, equidistant clusters, because the loss encourages every wrong class to sit at roughly the same distance from the correct one. This geometry helps classification but erases information about class similarity.

That erased information is exactly what a student needs during distillation. For this reason, teachers trained with label smoothing often produce weaker students, even when the teacher itself is more accurate. The effect is a well-known trade-off rather than a bug.

When to apply label smoothing and when to avoid it

Label smoothing is most useful when labels are clean but the model is large enough to memorize them, and least useful when downstream steps depend on the raw shape of the output distribution. The decision table below summarizes common situations.

SituationRecommendationReason
Large image classifierUse epsilon around 0.1Reduces overconfidence, small accuracy gain
Transformer translation modelUse epsilon around 0.1Improves BLEU despite higher perplexity
Teacher for later distillationAvoid or keep very lowPreserves class similarity
Noisy or crowdsourced labelsConsider moderate smoothingSoftens the cost of wrong labels
Model used for out-of-distribution detectionEvaluate carefullyConfidence scores become less separable
Small number of classesUse lower valuesEach wrong class receives more mass
Decision guide for label smoothing

A frequent observation in translation is that label smoothing raises perplexity while improving task metrics. This is expected: the model deliberately assigns less probability to the reference token, so likelihood-based metrics look worse even though decoded outputs improve.

Common mistakes and limitations

Most problems with label smoothing come from applying it automatically without checking its interactions. The following points recur in practice.

  • Double smoothing: applying smoothing in the loss while the data already contains soft labels compounds the effect.
  • Ignoring padding tokens: in sequence models, smoothing mass placed on padding or special tokens distorts the target.
  • Comparing perplexity across settings: smoothed and unsmoothed models cannot be compared fairly on likelihood alone.
  • Expecting robustness to heavy label noise: smoothing helps a little, but dedicated noise-robust losses work better.
  • Using it during preference optimization without thought: methods such as DPO already have their own regularization terms.

Label smoothing is cheap, adding negligible compute cost, and it rarely damages accuracy at standard settings. Its limitation is that its benefits are modest and task-dependent, so it should be validated on a held-out set like any other regularizer.

Understanding it simply

Imagine a teacher who marks every answer as either completely right or completely wrong. A student trained this way may become stubbornly overconfident, insisting on answers even when unsure. Label smoothing is like a teacher who says, “This answer is correct, but keep a little doubt in mind.”

During training, the computer is told that the right answer deserves about 90 percent of its belief instead of 100 percent. That small dose of humility keeps the model from becoming too sure of itself, which usually makes its confidence more honest and its results slightly better on new examples.

Applied in practice

A team building a model that sorts product photos into 400 categories for an online catalog noticed that the classifier reported confidence above 98 percent even on images it got wrong. That made it impossible to route uncertain images to human reviewers, because nearly every prediction looked certain.

The engineers retrained the same network with label smoothing at epsilon 0.1, keeping all other settings unchanged. Top-1 accuracy rose by a fraction of a percentage point, but the larger change was in calibration: the expected calibration error fell substantially, and wrong predictions now clustered at noticeably lower confidence.

With a confidence threshold of 0.7, the team sent about 8 percent of images to human review and caught most of the classification errors there. The outcome was a more trustworthy review queue at almost no extra training cost. When the team later planned to distill the model into a smaller version for mobile devices, they trained a separate unsmoothed teacher to preserve class similarity information.