The learning rate is the hyperparameter that sets the size of each parameter update in gradient-based optimization: after the gradient of the loss function is computed, the weights move in the opposite direction by an amount scaled by the learning rate. In the training of neural networks, from small classifiers to large language models, it is widely regarded as the single most influential hyperparameter, because it decides whether training converges, how fast, and to what quality.
A learning rate that is too high makes the loss oscillate or explode; one that is too low wastes compute and may leave the model stuck in a poor region. In practice the learning rate is rarely a constant. It follows a schedule that changes its value over the course of training, and adaptive optimizers rescale it separately for each parameter.
How the learning rate shapes gradient descent
In plain gradient descent, each step updates the weights as new weights equal old weights minus the learning rate times the gradient. The gradient gives the direction of steepest increase of the loss; the learning rate, usually written as eta or alpha, gives the step length. Because the gradient is estimated from a mini-batch, each step is also noisy, and the learning rate controls how strongly that noise is amplified.
The behavior of training changes in recognizable ways across the range of learning rates, as summarized in the table.
| Learning rate level | Loss curve behavior | Typical outcome |
|---|---|---|
| Far too high | Loss jumps upward or becomes NaN | Training diverges |
| Slightly too high | Loss falls, then oscillates or spikes | Unstable, worse final quality |
| Well chosen | Fast, steady decrease | Good convergence within budget |
| Too low | Very slow, almost flat decrease | Underfitting or wasted compute |
There is a link between the learning rate and the curvature of the loss surface. When the step is larger than the surface can tolerate in its steepest direction, updates overshoot the valley and bounce back and forth, which is why instability usually appears suddenly once a threshold is crossed.
Typical values by optimizer and setting
Useful learning rates depend heavily on the optimizer, model size and batch size. Values below are common starting points, not universal rules, and they assume a schedule with warmup and decay.
| Setting | Typical learning rate | Comment |
|---|---|---|
| SGD with momentum, image models | About 0.01 to 0.1 | Often scaled with batch size |
| Adam or AdamW, small networks | About 1e-4 to 1e-3 | 3e-4 is a frequent default |
| Pretraining large transformers | About 1e-4 to 6e-4 peak | Smaller for larger models |
| Full fine-tuning of a pretrained model | About 1e-6 to 5e-5 | Low to protect learned knowledge |
| LoRA or QLoRA adapters | About 1e-4 to 3e-4 | Higher because only adapters train |
With larger batches, gradients are less noisy and the learning rate can usually be increased. A common heuristic for SGD is the linear scaling rule, which raises the rate in proportion to the batch size; for adaptive optimizers a square-root scaling is often closer to practice. Both heuristics break down at very large batch sizes.
Learning rate schedules
A learning rate schedule changes the rate during training because the needs of early and late training differ. Early on, large steps make fast progress, but the very first steps on random weights are fragile, so a warmup phase ramps the rate up from near zero. Later, smaller steps let the model settle into a good minimum, which is the purpose of decay.
| Schedule | Shape over training | Where it is used |
|---|---|---|
| Constant | Fixed value throughout | Quick experiments, some fine-tuning |
| Step decay | Drops by a factor at set epochs | Classic image classification |
| Linear warmup with cosine decay | Rise, then smooth fall to a floor | Most transformer training |
| Warmup, stable, decay | Rise, long plateau, short final decay | Long or extendable pretraining runs |
| One-cycle | Rise to a high peak, then fall below start | Fast training of smaller models |
| Reduce on plateau | Drops when validation stops improving | Adaptive, data-driven training |
Warmup typically lasts from a few hundred to a few thousand steps, or a small percentage of total steps. Cosine schedules usually decay to about 10 percent of the peak or close to zero, depending on whether training may continue afterward.
Learning rate and neighboring concepts
The learning rate interacts closely with other training settings, and confusing them leads to misdiagnosis. It is a hyperparameter, set before or during training rather than learned from data, and it is distinct from the gradient itself, from the number of epochs and from regularization strength.
| Related setting | Interaction | Practical consequence |
|---|---|---|
| Batch size | Larger batches tolerate larger rates | Retune the rate when batch size changes |
| Momentum | Accumulates past updates | Effective step exceeds the nominal rate |
| Weight decay | Applied relative to the rate in AdamW | Changing the rate changes regularization |
| Gradient clipping | Caps the gradient norm | Protects against spikes at high rates |
| Number of epochs | Fixes the schedule length | Shorter runs often need higher peaks |
Finding a good learning rate and common mistakes
The most reliable method is a short sweep on a logarithmic grid, such as 1e-5, 3e-5, 1e-4, 3e-4 and 1e-3, comparing the loss after a fixed number of steps. A learning rate range test offers a faster estimate: the rate grows exponentially over a few hundred steps, and a good peak lies somewhat below the value at which the loss begins to rise.
- Reusing a rate after changing the model or batch size often leads to divergence or slow training.
- Skipping warmup with adaptive optimizers on transformers frequently causes early instability.
- Fine-tuning with pretraining-level rates can erase useful knowledge, a form of catastrophic forgetting.
- Reading a single noisy run as a trend; comparisons need fixed seeds or several runs.
- Ignoring loss spikes: repeated spikes usually call for a lower peak or stronger clipping.
The learning rate should be logged together with the loss at every step, since a plot of both curves is often the quickest way to see whether a problem comes from the schedule or from the data.
The idea in brief
Training a model is like walking downhill in fog toward the lowest point of a valley. The learning rate is the length of each step. Huge steps may jump over the valley and up the other side. Tiny steps are safe but painfully slow.
Good practice is to start gently, stride out, then slow down near the bottom. That pattern is what a learning rate schedule does automatically.
A practical scenario
A data scientist fine-tunes a pretrained text classifier on 50,000 support tickets using AdamW. The first run uses a learning rate of 1e-3, copied from a script for training a small network from scratch. Validation accuracy drops below that of the original model, and the loss curve shows sharp spikes.
She reruns a sweep at 1e-5, 2e-5 and 5e-5 with linear warmup over 6 percent of steps and linear decay. The run at 2e-5 gives the best validation accuracy and a smooth loss curve. Because each run takes under an hour, the sweep costs one afternoon and produces a model that is clearly better than the original baseline, without any change to data or architecture.