An optimizer is the algorithm that decides how model parameters change after each training step, turning the gradient of the loss function into a concrete weight update. In artificial intelligence, the optimizer sits at the center of every training loop: backpropagation computes which direction reduces the error, and the optimizer decides how far to move and in which scaled direction.
Training a modern network with billions of parameters is only practical because optimizers such as SGD with momentum, Adam and AdamW convert noisy mini-batch gradients into stable progress. The choice of optimizer influences convergence speed, final accuracy, memory consumption and how sensitive a run is to its hyperparameters.
How an optimizer turns gradients into weight updates
An optimizer applies an update rule that maps the current parameters, the current gradient and its own internal state to new parameters. The simplest rule subtracts the gradient multiplied by the learning rate; more advanced rules keep running statistics of past gradients and use them to rescale each step.
Each training step follows the same cycle. The model processes a mini-batch, the loss is computed, backpropagation produces a gradient for every parameter, and the optimizer performs the parameter update before the gradients are cleared for the next batch.
The main building blocks of optimizer state are listed below, together with what each one contributes to the update.
| Component | What it stores or does | Effect on training |
|---|---|---|
| Learning rate | Global scale of every step | Sets speed and stability |
| Momentum buffer | Exponential average of past gradients | Smooths noise, accelerates along consistent directions |
| Second-moment estimate | Average of squared gradients per parameter | Gives each parameter its own effective step size |
| Bias correction | Rescales early moment estimates | Prevents overly small or large first steps |
| Weight decay term | Shrinks weights toward zero | Acts as regularization |
| Epsilon | Small constant in the denominator | Avoids division by zero, limits extreme steps |
Major optimizer families and their update rules
Optimizers fall into a few families that differ in how much gradient history they keep and whether they adapt the step size per parameter. Plain stochastic gradient descent uses only the current gradient, momentum methods add a velocity term, and adaptive methods divide each step by an estimate of gradient magnitude.
The following comparison summarizes the optimizers most often encountered in deep learning practice.
| Optimizer | Key idea | Extra state per parameter | Typical use |
|---|---|---|---|
| SGD | Step against the raw gradient | None | Simple baselines, convex problems |
| SGD with momentum | Accumulate a velocity vector | One value | Image classification, convolutional networks |
| RMSProp | Divide by running root mean square | One value | Recurrent networks, reinforcement learning |
| Adam | Momentum plus per-parameter scaling | Two values | General default for deep learning |
| AdamW | Adam with decoupled weight decay | Two values | Transformers and large language models |
| Adafactor | Factorized second-moment estimate | Less than one value | Memory-constrained large models |
| Lion | Sign of an interpolated momentum | One value | Experimental large-scale training |
AdamW is the standard choice for transformer pretraining and fine-tuning because decoupled weight decay regularizes consistently regardless of gradient scale. Typical settings are beta1 of 0.9, beta2 between 0.95 and 0.999 and epsilon near 1e-8, although large runs often lower beta2 to improve stability.
Optimizer hyperparameters and what they control
Optimizer hyperparameters determine how aggressively and how smoothly the model moves through the loss landscape. The learning rate dominates all others, but the momentum coefficients and weight decay can decide whether a long run converges or diverges.
| Hyperparameter | Typical range | What happens if too high | What happens if too low |
|---|---|---|---|
| Learning rate | 1e-5 to 1e-1, depending on optimizer | Divergence or loss spikes | Slow progress, wasted compute |
| Beta1 (momentum) | 0.8 to 0.95 | Overshooting, sluggish reaction | Noisy, jittery updates |
| Beta2 | 0.95 to 0.999 | Slow adaptation to gradient changes | Unstable per-parameter scaling |
| Weight decay | 0 to 0.1 | Underfitting | Weaker regularization |
| Epsilon | 1e-8 to 1e-6 | Behaves more like SGD | Possible numerical instability |
The optimizer rarely runs with a fixed learning rate. A learning rate schedule with a short warmup phase followed by cosine or linear decay is standard, and gradient clipping at a global norm of about 1.0 commonly protects adaptive optimizers from rare extreme gradients.
Choosing an optimizer for a training task
The right optimizer depends on model architecture, scale and memory budget rather than on a universal ranking. Published benchmarks show that, once each optimizer is carefully tuned, differences in final quality are often small, so tuning effort and robustness matter more than the name of the algorithm.
| Situation | Recommended choice | Reason |
|---|---|---|
| Transformer pretraining | AdamW with warmup and cosine decay | Robust to sparse, uneven gradients |
| Fine-tuning a pretrained model | AdamW with a small learning rate | Preserves pretrained knowledge |
| Convolutional image model | SGD with momentum | Often generalizes slightly better |
| Tight GPU memory | Adafactor or 8-bit Adam | Reduces optimizer state size |
| Small tabular network | Adam with default settings | Fast convergence with little tuning |
Memory cost is a practical constraint that is easy to underestimate. Adam keeps two extra values per parameter, usually in 32-bit precision, so a 7-billion-parameter model needs about 56 GB for optimizer state alone. Sharded optimizers, 8-bit state and parameter-efficient methods such as LoRA exist largely to reduce this overhead.
Limitations and common mistakes
Optimizer problems usually appear as unstable loss curves, stalled progress or poor generalization, and they are frequently caused by configuration rather than by the algorithm itself.
- Reusing a learning rate across optimizers fails because Adam and SGD operate at very different scales; a good SGD rate is often far too large for Adam.
- Using L2 regularization inside Adam instead of decoupled weight decay weakens regularization for parameters with large gradients.
- Skipping warmup with adaptive optimizers can cause early divergence, because second-moment estimates are unreliable in the first steps.
- Resetting optimizer state when resuming from a checkpoint produces a sudden loss jump; the moment buffers must be saved and restored.
- Applying weight decay to biases and normalization layers is generally avoided, since it can degrade quality without useful regularization.
A further limitation is that no optimizer guarantees a global minimum in non-convex deep networks. The optimizer finds a good region of the loss landscape, and the result still depends on initialization, data order, batch size and the schedule.
A plain-language summary
An optimizer is the part of training that actually changes the model. After every batch of examples, the model learns which way its mistakes point, and the optimizer decides how big a correction to make and where.
Think of a hiker walking downhill in fog. The slope under the feet shows the direction, while the optimizer is the hiker’s judgment about step length and pace: careful steps on uncertain ground, longer strides on a steady slope. A good optimizer reaches the valley quickly without stumbling or overshooting.
An example from practice
A machine learning team fine-tunes a 1-billion-parameter language model on customer support transcripts. The first attempt uses Adam at a learning rate of 1e-3 without warmup, and the loss spikes and diverges within a few hundred steps.
The team switches to AdamW with a learning rate of 2e-5, a warmup of 500 steps, cosine decay, weight decay of 0.01 excluded from biases and normalization layers, and gradient clipping at 1.0. Because GPU memory is limited, they also enable an 8-bit optimizer state, cutting optimizer memory by roughly three quarters.
The second run trains smoothly for three epochs, validation loss falls steadily, and the resulting model answers support questions noticeably better than the base model. The optimizer configuration, not the data or architecture, made the difference between a failed and a successful run.