Optimizer

On this page 7

An optimizer is the algorithm that decides how model parameters change after each training step, turning the gradient of the loss function into a concrete weight update. In artificial intelligence, the optimizer sits at the center of every training loop: backpropagation computes which direction reduces the error, and the optimizer decides how far to move and in which scaled direction.

Training a modern network with billions of parameters is only practical because optimizers such as SGD with momentum, Adam and AdamW convert noisy mini-batch gradients into stable progress. The choice of optimizer influences convergence speed, final accuracy, memory consumption and how sensitive a run is to its hyperparameters.

How an optimizer turns gradients into weight updates

An optimizer applies an update rule that maps the current parameters, the current gradient and its own internal state to new parameters. The simplest rule subtracts the gradient multiplied by the learning rate; more advanced rules keep running statistics of past gradients and use them to rescale each step.

Each training step follows the same cycle. The model processes a mini-batch, the loss is computed, backpropagation produces a gradient for every parameter, and the optimizer performs the parameter update before the gradients are cleared for the next batch.

The main building blocks of optimizer state are listed below, together with what each one contributes to the update.

ComponentWhat it stores or doesEffect on training
Learning rateGlobal scale of every stepSets speed and stability
Momentum bufferExponential average of past gradientsSmooths noise, accelerates along consistent directions
Second-moment estimateAverage of squared gradients per parameterGives each parameter its own effective step size
Bias correctionRescales early moment estimatesPrevents overly small or large first steps
Weight decay termShrinks weights toward zeroActs as regularization
EpsilonSmall constant in the denominatorAvoids division by zero, limits extreme steps
Core components of a gradient-based optimizer

Major optimizer families and their update rules

Optimizers fall into a few families that differ in how much gradient history they keep and whether they adapt the step size per parameter. Plain stochastic gradient descent uses only the current gradient, momentum methods add a velocity term, and adaptive methods divide each step by an estimate of gradient magnitude.

The following comparison summarizes the optimizers most often encountered in deep learning practice.

OptimizerKey ideaExtra state per parameterTypical use
SGDStep against the raw gradientNoneSimple baselines, convex problems
SGD with momentumAccumulate a velocity vectorOne valueImage classification, convolutional networks
RMSPropDivide by running root mean squareOne valueRecurrent networks, reinforcement learning
AdamMomentum plus per-parameter scalingTwo valuesGeneral default for deep learning
AdamWAdam with decoupled weight decayTwo valuesTransformers and large language models
AdafactorFactorized second-moment estimateLess than one valueMemory-constrained large models
LionSign of an interpolated momentumOne valueExperimental large-scale training
Common optimizers compared

AdamW is the standard choice for transformer pretraining and fine-tuning because decoupled weight decay regularizes consistently regardless of gradient scale. Typical settings are beta1 of 0.9, beta2 between 0.95 and 0.999 and epsilon near 1e-8, although large runs often lower beta2 to improve stability.

Optimizer hyperparameters and what they control

Optimizer hyperparameters determine how aggressively and how smoothly the model moves through the loss landscape. The learning rate dominates all others, but the momentum coefficients and weight decay can decide whether a long run converges or diverges.

HyperparameterTypical rangeWhat happens if too highWhat happens if too low
Learning rate1e-5 to 1e-1, depending on optimizerDivergence or loss spikesSlow progress, wasted compute
Beta1 (momentum)0.8 to 0.95Overshooting, sluggish reactionNoisy, jittery updates
Beta20.95 to 0.999Slow adaptation to gradient changesUnstable per-parameter scaling
Weight decay0 to 0.1UnderfittingWeaker regularization
Epsilon1e-8 to 1e-6Behaves more like SGDPossible numerical instability
Key optimizer hyperparameters

The optimizer rarely runs with a fixed learning rate. A learning rate schedule with a short warmup phase followed by cosine or linear decay is standard, and gradient clipping at a global norm of about 1.0 commonly protects adaptive optimizers from rare extreme gradients.

Choosing an optimizer for a training task

The right optimizer depends on model architecture, scale and memory budget rather than on a universal ranking. Published benchmarks show that, once each optimizer is carefully tuned, differences in final quality are often small, so tuning effort and robustness matter more than the name of the algorithm.

SituationRecommended choiceReason
Transformer pretrainingAdamW with warmup and cosine decayRobust to sparse, uneven gradients
Fine-tuning a pretrained modelAdamW with a small learning ratePreserves pretrained knowledge
Convolutional image modelSGD with momentumOften generalizes slightly better
Tight GPU memoryAdafactor or 8-bit AdamReduces optimizer state size
Small tabular networkAdam with default settingsFast convergence with little tuning
Optimizer selection by situation

Memory cost is a practical constraint that is easy to underestimate. Adam keeps two extra values per parameter, usually in 32-bit precision, so a 7-billion-parameter model needs about 56 GB for optimizer state alone. Sharded optimizers, 8-bit state and parameter-efficient methods such as LoRA exist largely to reduce this overhead.

Limitations and common mistakes

Optimizer problems usually appear as unstable loss curves, stalled progress or poor generalization, and they are frequently caused by configuration rather than by the algorithm itself.

  • Reusing a learning rate across optimizers fails because Adam and SGD operate at very different scales; a good SGD rate is often far too large for Adam.
  • Using L2 regularization inside Adam instead of decoupled weight decay weakens regularization for parameters with large gradients.
  • Skipping warmup with adaptive optimizers can cause early divergence, because second-moment estimates are unreliable in the first steps.
  • Resetting optimizer state when resuming from a checkpoint produces a sudden loss jump; the moment buffers must be saved and restored.
  • Applying weight decay to biases and normalization layers is generally avoided, since it can degrade quality without useful regularization.

A further limitation is that no optimizer guarantees a global minimum in non-convex deep networks. The optimizer finds a good region of the loss landscape, and the result still depends on initialization, data order, batch size and the schedule.

A plain-language summary

An optimizer is the part of training that actually changes the model. After every batch of examples, the model learns which way its mistakes point, and the optimizer decides how big a correction to make and where.

Think of a hiker walking downhill in fog. The slope under the feet shows the direction, while the optimizer is the hiker’s judgment about step length and pace: careful steps on uncertain ground, longer strides on a steady slope. A good optimizer reaches the valley quickly without stumbling or overshooting.

An example from practice

A machine learning team fine-tunes a 1-billion-parameter language model on customer support transcripts. The first attempt uses Adam at a learning rate of 1e-3 without warmup, and the loss spikes and diverges within a few hundred steps.

The team switches to AdamW with a learning rate of 2e-5, a warmup of 500 steps, cosine decay, weight decay of 0.01 excluded from biases and normalization layers, and gradient clipping at 1.0. Because GPU memory is limited, they also enable an 8-bit optimizer state, cutting optimizer memory by roughly three quarters.

The second run trains smoothly for three epochs, validation loss falls steadily, and the resulting model answers support questions noticeably better than the base model. The optimizer configuration, not the data or architecture, made the difference between a failed and a successful run.