Weight decay

On this page 7

Weight decay is a regularization technique that shrinks every trainable weight of a neural network by a small fraction at each optimization step, so that parameters stay small unless the data keeps pushing them up. In artificial intelligence, it is one of the standard controls of Training, used in almost every modern recipe from image classifiers to large language models.

The method rests on a simple idea: models with smaller, smoother weights tend to generalize better than models with large, finely tuned ones. Weight decay expresses that preference directly in the update rule and is set as a hyperparameter, usually written as lambda or weight_decay in training code.

How weight decay modifies each update

Weight decay works by multiplying each weight by a factor slightly below one before or alongside the gradient update. In plain stochastic gradient descent, the update becomes w ← w − η(∇L + λw), where η is the learning rate and λ is the decay coefficient. The term ηλw pulls every weight toward zero in proportion to its current size, so large weights shrink faster than small ones.

For plain SGD, this rule is mathematically identical to adding an L2 penalty of (λ/2)·‖w‖² to the loss function. That equivalence explains why the two names are often used interchangeably. It also explains the effect: the optimizer now balances fitting the data against keeping the squared norm of the weights low, which discourages the model from relying on a few extreme parameters.

The following table summarizes the quantities that determine how strong the shrinkage is in practice.

QuantityRoleTypical range
Decay coefficient λSets the pull toward zero1e-5 to 1e-4 (SGD), 0.01 to 0.1 (AdamW)
Learning rate ηScales the shrink per step1e-5 to 1e-1, model dependent
Effective shrink ηλFraction removed each stepOften 1e-7 to 1e-5
Number of stepsAccumulates total shrinkageThousands to millions
Quantities that control the strength of weight decay

Coupled L2 regularization versus decoupled weight decay

The equivalence with L2 regularization breaks down for adaptive optimizers such as the Adam optimizer. Adam divides each gradient by a running estimate of its magnitude. If the L2 penalty is added to the gradient, the penalty is also rescaled, so weights with large historical gradients are regularized less than intended.

The fix, published as AdamW in 2017, is decoupled weight decay: the shrink step is applied directly to the weights, outside the adaptive rescaling. Every parameter then decays at the same relative rate regardless of its gradient history. Decoupled decay generalizes measurably better in many settings and has become the default in transformer training.

The table below contrasts the two formulations that practitioners most often confuse.

AspectCoupled L2 penaltyDecoupled weight decay
Where it actsAdded to the gradientApplied directly to weights
Interaction with AdamRescaled by adaptive termsIndependent of adaptive terms
Typical λ with AdamVery small, hard to tune0.01 to 0.1
Common implementationAdam with weight_decay argumentAdamW
Same as L2 for SGDYesYes, up to learning rate scaling
Two ways to implement weight decay

A frequent source of confusion is that some libraries expose a weight_decay argument on plain Adam that implements the coupled penalty, while AdamW implements the decoupled version. The same numeric value therefore produces very different behavior depending on the optimizer class.

Why weight decay improves generalization

Weight decay improves generalization mainly by limiting effective model capacity. Small weights produce smoother functions whose outputs change gradually with the input, which makes them less likely to memorize noise in the training set. This links weight decay to classic arguments about overfitting and model complexity.

In networks with normalization layers, the mechanism is subtler. Layers followed by batch or layer normalization are scale invariant, so shrinking their weights does not change the output. Here weight decay instead raises the effective learning rate by keeping the weight norm small, which changes optimization dynamics more than capacity. Research on this interaction is active, and explanations remain partly debated.

In large language model pretraining, weight decay also stabilizes long runs. Unchecked weight growth can contribute to loss spikes and to numerical problems in mixed-precision training, and a moderate decay of about 0.1 with AdamW is a widely used default.

Choosing and configuring weight decay

Weight decay should be tuned jointly with the learning rate, because the effective shrink per step is their product. A common practice is to keep λ fixed while sweeping the learning rate, then adjust λ in factors of three to ten. The decision table below lists typical starting points.

SetupRecommended choiceReason
CNN with SGD and momentumL2 or decay of 5e-4 to 1e-4Long established baseline
Transformer pretrainingAdamW with λ of 0.1Stable, well tested default
Fine-tuning a pretrained modelAdamW with λ of 0.0 to 0.01Avoid pulling weights from good values
LoRA adaptersSmall or zero decay on adaptersFew parameters, low overfitting risk
Small dataset, large modelHigher λ plus early stoppingStrong overfitting pressure
Starting points for common training setups

Most frameworks let the practitioner build parameter groups. A standard rule is to exclude biases, normalization gains and often embeddings from decay, since shrinking them adds little regularization and can hurt accuracy. Decay should also follow the learning rate schedule, which AdamW does automatically when the decay step is multiplied by the scheduled rate.

Limitations and common mistakes

Weight decay is not a substitute for adequate data or proper validation. It reduces variance but cannot fix label noise, distribution shift or a flawed loss function. The table below lists frequent errors and their fixes.

SymptomLikely causeFix
Training loss stalls highλ far too largeReduce λ by a factor of ten
Large train and validation gapλ too small or zeroIncrease λ, add data augmentation
No effect from λ with AdamCoupled penalty rescaled awaySwitch to AdamW
Accuracy drops after fine-tuningDecay erodes pretrained weightsLower λ or decay toward initial weights
Unstable normalization layersDecay applied to gainsExclude normalization parameters
Frequent weight decay errors

Another misconception is that weight decay and dropout are interchangeable. Dropout injects noise into activations, while weight decay constrains parameter magnitude; they address overfitting through different mechanisms and are often combined. Finally, results reported with a given λ are not portable across optimizers or batch sizes without retuning.

The short version

A neural network learns by adjusting millions of numbers called weights. If those numbers grow too large, the model starts to memorize its training examples instead of learning general patterns. Weight decay gently shrinks every weight a little after each learning step, the way a small tax trims every balance.

Weights that the data truly needs keep getting pushed back up, so they survive. Weights that only help with odd, noisy examples fade away. The result is a simpler model that usually performs better on new data it has never seen.

Where it makes a difference

A machine learning team fine-tunes a vision transformer to classify defects on a production line, using only 4,000 labeled images. With AdamW and weight decay set to zero, training accuracy reaches 99 percent after ten epochs, but validation accuracy stalls at 86 percent and then starts to fall, a clear sign of overfitting.

The team sets λ to 0.05, excludes biases and layer normalization parameters from decay, and keeps the same learning rate schedule. Training accuracy now peaks at 96 percent, while validation accuracy rises to 91 percent and stays stable across later epochs. The smaller gap means the deployed model misclassifies fewer unusual parts on the real line, at no extra computing cost.