Adam optimizer

Also known as: Adam

On this page 7

The Adam optimizer (short for adaptive moment estimation) is a gradient-based optimization algorithm that updates each model parameter with its own step size, computed from exponential moving averages of the gradient and of the squared gradient. In artificial intelligence, Adam and its variant AdamW are the default choice for training most deep neural networks, including transformers and large language models.

Training with Adam combines two older ideas: momentum, which smooths the update direction, and per-parameter scaling in the style of RMSProp, which shrinks steps for parameters with large or noisy gradients. The result is an optimizer that works with little manual tuning across very different architectures, at the cost of extra memory and some subtle generalization trade-offs.

How the Adam update rule works

Adam keeps two running statistics for every parameter and uses them to form a normalized, bias-corrected step. At each step t, the optimizer reads the current gradient g from backpropagation and updates its state before changing the weight.

  1. First moment: m = beta1 * m + (1 – beta1) * g, a running mean of the gradient that acts as momentum.
  2. Second moment: v = beta2 * v + (1 – beta2) * g^2, a running mean of the squared gradient that estimates its scale.
  3. Bias correction: m_hat = m / (1 – beta1^t) and v_hat = v / (1 – beta2^t), because both averages start at zero and would otherwise be too small in early steps.
  4. Parameter update: w = w – lr * m_hat / (sqrt(v_hat) + epsilon).

The division by the square root of v_hat is the core mechanism. A parameter whose gradients are consistently large gets a smaller effective step, while a parameter with rare or small gradients gets a relatively larger one. Because the ratio m_hat / sqrt(v_hat) is roughly bounded by one, the magnitude of each update is controlled mainly by the learning rate, which makes the step size invariant to gradient rescaling.

The following table lists the state Adam stores and why each piece matters in practice.

StateWhat it estimatesPractical effect
First moment mMean direction of recent gradientsDamps oscillation, speeds progress in consistent directions
Second moment vMean squared gradient magnitudeScales each parameter’s step individually
Step counter tNumber of updates performedDrives bias correction in early training
Model weightsCurrent parameter valuesThe quantity being optimized
Internal state maintained by Adam

Adam hyperparameters and typical values

Adam has four hyperparameters, and the published defaults work surprisingly often, which is a large part of its popularity. In large-scale training, however, beta2 and epsilon are frequently adjusted to improve numerical stability.

HyperparameterCommon defaultTypical rangeWhat it controls
Learning rate (lr)0.0011e-5 to 3e-3Overall step magnitude
beta10.90.8 to 0.95Memory of the momentum average
beta20.9990.95 to 0.999Memory of the squared-gradient average
epsilon1e-81e-8 to 1e-6Prevents division by zero, caps steps
Adam hyperparameters with common ranges

Lower beta2 values such as 0.95 are common when pretraining large language models, because a shorter memory lets the second moment react faster to sudden gradient changes and reduces the risk of a loss spike. A larger epsilon makes Adam behave more like momentum SGD for parameters with tiny gradients. Learning rates for transformers are usually far below the 0.001 default and are paired with a warmup phase and a decaying learning rate schedule.

Adam belongs to a family of adaptive optimizers that sit between plain stochastic gradient descent and second-order methods. The comparison below shows where it differs from its neighbors and successors.

OptimizerPer-parameter scalingMomentumMemory per parameterTypical use
SGD with momentumNoYesOne extra valueVision models, strong generalization
RMSPropYesOptionalOne extra valueRecurrent networks, reinforcement learning
AdamYesYesTwo extra valuesGeneral deep learning default
AdamWYesYesTwo extra valuesTransformers and language models
AdafactorFactored approximationOptionalSublinearMemory-constrained large models
Adam and neighboring optimization methods

The most important distinction is between Adam and AdamW. In original Adam, L2 regularization is added to the gradient, so it is also divided by sqrt(v_hat) and becomes weak for parameters with large gradients. AdamW decouples weight decay by subtracting lr * lambda * w directly from the weights, which restores the intended regularization and generally improves generalization. Most modern training code that says “Adam” in practice runs AdamW.

Using Adam in deep learning practice

Adam is a strong first choice for new architectures, because it converges quickly without careful per-layer tuning. The decision table below summarizes common situations and reasonable starting configurations.

SituationRecommended setupReason
Prototyping a new modelAdam, lr 1e-3, default betasFast, reliable convergence with minimal tuning
Transformer pretrainingAdamW, beta2 0.95, warmup plus cosine decayStability under large batches and long runs
Fine-tuning a pretrained modelAdamW, lr 1e-5 to 5e-5Avoids overwriting pretrained knowledge
Image classification at scaleCompare AdamW with SGD with momentumSGD sometimes generalizes better
Limited GPU memory8-bit Adam or AdafactorReduces optimizer state size
Choosing an Adam configuration

Memory cost deserves attention. Adam stores two additional values per parameter, usually in 32-bit precision, so in mixed-precision training the optimizer state can exceed the size of the model weights themselves. Techniques such as 8-bit optimizer states, sharded optimizers and gradient checkpointing exist largely to manage this overhead in large models.

Limitations and common mistakes with Adam

Adam is robust but not foolproof, and several failure modes recur in practice. Knowing them helps distinguish an optimizer problem from a data or architecture problem.

  • Generalization gap: on some vision benchmarks, adaptive methods reach lower training loss but slightly worse test accuracy than well-tuned SGD.
  • Coupled weight decay: using L2 penalties with plain Adam instead of AdamW gives weaker and uneven regularization.
  • Instability at scale: large learning rates without warmup can cause divergence or recurring loss spikes.
  • Convergence theory gaps: the original proof contained an error, and AMSGrad was proposed to fix non-convergence on constructed examples, although it rarely helps in practice.
  • Resetting optimizer state: restarting a run from a checkpoint without the saved m and v values causes a temporary jump in loss.

When training misbehaves, practitioners usually check the learning rate first, then warmup length, beta2 and gradient clipping, before considering a different optimizer.

The simple explanation

Training a neural network means nudging millions of internal numbers again and again until the model makes fewer mistakes. Adam decides how big each nudge should be, separately for every number.

Think of hiking downhill in fog. Adam remembers which way you have been heading, so you keep moving smoothly instead of zigzagging. It also notices where the ground is very bumpy and takes smaller, more careful steps there, while striding faster on gentle slopes. That combination is why Adam usually works well without much fine adjustment, and why it became the standard tool for training modern AI models.

How it is used in practice

A machine learning team fine-tunes a 7-billion-parameter language model to answer customer support questions. It selects AdamW with a learning rate of 2e-5, beta1 of 0.9, beta2 of 0.95, weight decay of 0.1, a warmup of 100 steps and a cosine decay over three epochs.

The first attempt used a learning rate of 1e-4 without warmup, and the loss spiked within a few hundred steps while the model began to lose general language ability. After lowering the rate and adding warmup, the loss decreased smoothly. To fit the optimizer state on the available GPUs, the team switched to 8-bit optimizer states, which cut memory use substantially with no measurable quality loss. The final model improved answer accuracy on the team’s internal evaluation set while keeping its general skills intact.