Momentum

On this page 7

Momentum is an optimization technique used in the training of machine learning models that updates parameters with a running average of past gradients instead of the current gradient alone. In artificial intelligence, it is a standard component of stochastic gradient descent and of adaptive optimizers such as Adam, where it damps oscillations and accelerates progress along consistent directions of the loss surface.

The idea comes from physics: a ball rolling downhill gathers velocity and keeps moving through small bumps. In model training, the accumulated velocity carries the parameters across flat regions and narrow valleys that would slow plain gradient descent, which is why nearly every large training run uses some form of momentum.

How momentum accumulates gradient history

Momentum maintains a velocity vector with the same shape as the model parameters and updates it at every step. In the classical formulation, the velocity is computed as v = beta * v + g, where g is the current mini-batch gradient and beta is the momentum coefficient, and the parameters then move by w = w - lr * v.

Because the velocity is an exponentially weighted sum of past gradients, recent gradients count most and older ones fade geometrically. With beta of 0.9, the effective memory spans roughly the last 10 steps, and in steady conditions the step size grows to about ten times the plain gradient step. Components of the gradient that point in the same direction step after step add up, while components that flip sign cancel out.

This filtering matters most on ill-conditioned loss surfaces, where curvature is steep in some directions and shallow in others. Plain gradient descent zigzags across the steep walls of such a valley; momentum averages out the zigzag and builds speed along the valley floor.

The table below shows how the coefficient sets the length of the gradient memory and the resulting amplification of a steady gradient.

Beta valueEffective memorySteady-state amplificationTypical use
0.01 step1xPlain SGD without momentum
0.5About 2 steps2xEarly warm-up or noisy tasks
0.9About 10 steps10xDefault for SGD with momentum
0.99About 100 steps100xVery smooth objectives, rarely used
Momentum coefficient and its effective memory

Classical, Nesterov and adaptive variants

Several formulations of momentum exist, and they differ in where the gradient is evaluated and how the velocity is normalized. Classical momentum, often called heavy-ball momentum after Polyak, evaluates the gradient at the current position. Nesterov accelerated gradient evaluates it at the point the velocity is about to carry the parameters to, which gives a corrective look-ahead and slightly better behavior near a minimum.

Adaptive optimizers reuse the same principle. In Adam and AdamW, the first-moment estimate is momentum with beta1 typically 0.9, combined with per-parameter scaling from a second-moment estimate. Deep learning frameworks also implement a dampening factor and an exponential moving average form, v = beta * v + (1 - beta) * g, which changes the effective learning rate by a factor of 1 / (1 - beta).

The following table compares the main variants found in current training code.

VariantGradient evaluated atVelocity formWhere it appears
Heavy-ball momentumCurrent parametersSum of past gradientsSGD with momentum
Nesterov momentumLook-ahead positionSum with corrected gradientSGD with nesterov flag
EMA momentumCurrent parametersWeighted average, scaled by 1 – betaAdam first moment
Momentum with bias correctionCurrent parametersAverage divided by 1 – beta^tAdam and AdamW
Common momentum variants

Momentum is often confused with neighboring ideas because all of them shape the update step. The key distinction is that momentum changes the direction and memory of the update, whereas the learning rate scales its size and weight decay pulls parameters toward zero. Momentum is not an optimizer in itself but a component inside optimizers.

ConceptWhat it controlsInteraction with momentum
Learning rateSize of each updateEffective step is about lr divided by 1 – beta
Batch sizeNoise level of each gradientLarger batches reduce the need for heavy averaging
Gradient accumulationGradients summed before one updateMomentum then averages across accumulated updates
Weight decayShrinkage of parametersDecoupled decay in AdamW bypasses the velocity
Gradient clippingMaximum gradient normClipped gradients enter the velocity, limiting spikes
How momentum differs from neighboring concepts

Because the effective step depends on both values, learning rate and momentum must be tuned together. Raising beta from 0.9 to 0.99 without lowering the learning rate multiplies the effective step size roughly tenfold, which frequently causes divergence.

Choosing momentum settings in practice

Most practitioners start from well-tested defaults and adjust momentum only when diagnostics point to it. For SGD in computer vision, a momentum of 0.9 with Nesterov enabled is a long-standing baseline. For transformer language models trained with AdamW, beta1 of 0.9 is the norm, while beta2 is commonly set to 0.95 for large runs to react faster to changes in gradient scale.

The decision table below summarizes typical situations and reasonable starting points.

SituationRecommended settingReason
Image classification with SGDBeta 0.9, Nesterov onProven baseline with good generalization
Large language model pretrainingAdamW beta1 0.9Stable first-moment estimate at scale
Very small batchesBeta 0.9 to 0.95Extra averaging offsets noisy gradients
Fine-tuning a pretrained modelDefaults with low learning rateMomentum rarely needs changes
Unstable early trainingLearning rate warm-upVelocity has not yet stabilized
Starting points for momentum by situation

Some schedules vary momentum during training. The one-cycle policy lowers momentum while the learning rate rises and raises it again as the learning rate falls, keeping the effective step size roughly balanced across the run.

Limitations and common mistakes

Momentum improves convergence speed on average, but it introduces overshoot risk and extra state. The velocity buffer uses one additional value per parameter, which matters for memory budgets in large models and is one reason memory-efficient optimizers compress or drop optimizer state.

  • Overshooting minima: a high beta keeps the parameters moving past a good region, producing oscillating loss curves.
  • Stale velocity after a restart: resuming from a checkpoint without the optimizer state resets momentum and can cause a loss spike.
  • Ignoring the coupling with learning rate: changing beta alone silently changes the effective step size.
  • Assuming momentum fixes bad conditioning: it helps, but normalization layers and adaptive scaling usually help more.
  • Confusing EMA and sum forms: porting hyperparameters between frameworks that implement different forms gives different effective rates.

In everyday language

Training a model means taking many small steps downhill on a landscape of errors. Without momentum, each step looks only at the slope right under its feet, so the model can wobble from side to side in narrow valleys and crawl across flat stretches.

Momentum lets the model remember the direction it has been moving. Like a sled that keeps sliding after a push, it speeds up when the slope stays consistent and smooths out small bumps. The result is usually faster and steadier learning, as long as the speed is not so high that it slides past the best spot.

Seen in the real world

A machine learning team trains a ResNet image classifier on a dataset of 1.2 million labeled photographs using plain stochastic gradient descent with a learning rate of 0.01. After 30 epochs, the validation accuracy is still climbing slowly, and the training loss curve shows a jagged, oscillating pattern.

The team switches to SGD with Nesterov momentum of 0.9, keeps the same learning rate schedule with a short warm-up, and saves optimizer state with every checkpoint. The loss curve becomes noticeably smoother, and the model reaches the earlier accuracy level in less than half the epochs. The saved compute lets the team run a longer schedule, which ends with a higher final accuracy at the same training budget.