Momentum is an optimization technique used in the training of machine learning models that updates parameters with a running average of past gradients instead of the current gradient alone. In artificial intelligence, it is a standard component of stochastic gradient descent and of adaptive optimizers such as Adam, where it damps oscillations and accelerates progress along consistent directions of the loss surface.
The idea comes from physics: a ball rolling downhill gathers velocity and keeps moving through small bumps. In model training, the accumulated velocity carries the parameters across flat regions and narrow valleys that would slow plain gradient descent, which is why nearly every large training run uses some form of momentum.
How momentum accumulates gradient history
Momentum maintains a velocity vector with the same shape as the model parameters and updates it at every step. In the classical formulation, the velocity is computed as v = beta * v + g, where g is the current mini-batch gradient and beta is the momentum coefficient, and the parameters then move by w = w - lr * v.
Because the velocity is an exponentially weighted sum of past gradients, recent gradients count most and older ones fade geometrically. With beta of 0.9, the effective memory spans roughly the last 10 steps, and in steady conditions the step size grows to about ten times the plain gradient step. Components of the gradient that point in the same direction step after step add up, while components that flip sign cancel out.
This filtering matters most on ill-conditioned loss surfaces, where curvature is steep in some directions and shallow in others. Plain gradient descent zigzags across the steep walls of such a valley; momentum averages out the zigzag and builds speed along the valley floor.
The table below shows how the coefficient sets the length of the gradient memory and the resulting amplification of a steady gradient.
| Beta value | Effective memory | Steady-state amplification | Typical use |
|---|---|---|---|
| 0.0 | 1 step | 1x | Plain SGD without momentum |
| 0.5 | About 2 steps | 2x | Early warm-up or noisy tasks |
| 0.9 | About 10 steps | 10x | Default for SGD with momentum |
| 0.99 | About 100 steps | 100x | Very smooth objectives, rarely used |
Classical, Nesterov and adaptive variants
Several formulations of momentum exist, and they differ in where the gradient is evaluated and how the velocity is normalized. Classical momentum, often called heavy-ball momentum after Polyak, evaluates the gradient at the current position. Nesterov accelerated gradient evaluates it at the point the velocity is about to carry the parameters to, which gives a corrective look-ahead and slightly better behavior near a minimum.
Adaptive optimizers reuse the same principle. In Adam and AdamW, the first-moment estimate is momentum with beta1 typically 0.9, combined with per-parameter scaling from a second-moment estimate. Deep learning frameworks also implement a dampening factor and an exponential moving average form, v = beta * v + (1 - beta) * g, which changes the effective learning rate by a factor of 1 / (1 - beta).
The following table compares the main variants found in current training code.
| Variant | Gradient evaluated at | Velocity form | Where it appears |
|---|---|---|---|
| Heavy-ball momentum | Current parameters | Sum of past gradients | SGD with momentum |
| Nesterov momentum | Look-ahead position | Sum with corrected gradient | SGD with nesterov flag |
| EMA momentum | Current parameters | Weighted average, scaled by 1 – beta | Adam first moment |
| Momentum with bias correction | Current parameters | Average divided by 1 – beta^t | Adam and AdamW |
Momentum compared with related training concepts
Momentum is often confused with neighboring ideas because all of them shape the update step. The key distinction is that momentum changes the direction and memory of the update, whereas the learning rate scales its size and weight decay pulls parameters toward zero. Momentum is not an optimizer in itself but a component inside optimizers.
| Concept | What it controls | Interaction with momentum |
|---|---|---|
| Learning rate | Size of each update | Effective step is about lr divided by 1 – beta |
| Batch size | Noise level of each gradient | Larger batches reduce the need for heavy averaging |
| Gradient accumulation | Gradients summed before one update | Momentum then averages across accumulated updates |
| Weight decay | Shrinkage of parameters | Decoupled decay in AdamW bypasses the velocity |
| Gradient clipping | Maximum gradient norm | Clipped gradients enter the velocity, limiting spikes |
Because the effective step depends on both values, learning rate and momentum must be tuned together. Raising beta from 0.9 to 0.99 without lowering the learning rate multiplies the effective step size roughly tenfold, which frequently causes divergence.
Choosing momentum settings in practice
Most practitioners start from well-tested defaults and adjust momentum only when diagnostics point to it. For SGD in computer vision, a momentum of 0.9 with Nesterov enabled is a long-standing baseline. For transformer language models trained with AdamW, beta1 of 0.9 is the norm, while beta2 is commonly set to 0.95 for large runs to react faster to changes in gradient scale.
The decision table below summarizes typical situations and reasonable starting points.
| Situation | Recommended setting | Reason |
|---|---|---|
| Image classification with SGD | Beta 0.9, Nesterov on | Proven baseline with good generalization |
| Large language model pretraining | AdamW beta1 0.9 | Stable first-moment estimate at scale |
| Very small batches | Beta 0.9 to 0.95 | Extra averaging offsets noisy gradients |
| Fine-tuning a pretrained model | Defaults with low learning rate | Momentum rarely needs changes |
| Unstable early training | Learning rate warm-up | Velocity has not yet stabilized |
Some schedules vary momentum during training. The one-cycle policy lowers momentum while the learning rate rises and raises it again as the learning rate falls, keeping the effective step size roughly balanced across the run.
Limitations and common mistakes
Momentum improves convergence speed on average, but it introduces overshoot risk and extra state. The velocity buffer uses one additional value per parameter, which matters for memory budgets in large models and is one reason memory-efficient optimizers compress or drop optimizer state.
- Overshooting minima: a high beta keeps the parameters moving past a good region, producing oscillating loss curves.
- Stale velocity after a restart: resuming from a checkpoint without the optimizer state resets momentum and can cause a loss spike.
- Ignoring the coupling with learning rate: changing beta alone silently changes the effective step size.
- Assuming momentum fixes bad conditioning: it helps, but normalization layers and adaptive scaling usually help more.
- Confusing EMA and sum forms: porting hyperparameters between frameworks that implement different forms gives different effective rates.
In everyday language
Training a model means taking many small steps downhill on a landscape of errors. Without momentum, each step looks only at the slope right under its feet, so the model can wobble from side to side in narrow valleys and crawl across flat stretches.
Momentum lets the model remember the direction it has been moving. Like a sled that keeps sliding after a push, it speeds up when the slope stays consistent and smooths out small bumps. The result is usually faster and steadier learning, as long as the speed is not so high that it slides past the best spot.
Seen in the real world
A machine learning team trains a ResNet image classifier on a dataset of 1.2 million labeled photographs using plain stochastic gradient descent with a learning rate of 0.01. After 30 epochs, the validation accuracy is still climbing slowly, and the training loss curve shows a jagged, oscillating pattern.
The team switches to SGD with Nesterov momentum of 0.9, keeps the same learning rate schedule with a short warm-up, and saves optimizer state with every checkpoint. The loss curve becomes noticeably smoother, and the model reaches the earlier accuracy level in less than half the epochs. The saved compute lets the team run a longer schedule, which ends with a higher final accuracy at the same training budget.