A learning rate schedule is a rule that changes the learning rate over the course of training, typically as a function of the current step or epoch, instead of holding it at one fixed value. Training of modern artificial intelligence models, from image classifiers to large language models, almost always relies on a schedule, because the step size that works best at the start of optimization is rarely the one that works best near the end.
Also called a learning rate scheduler or, when it only reduces the value, learning rate decay, the schedule is one of the most influential hyperparameters in gradient-based optimization. A poorly chosen schedule can cause loss spikes, divergence or a wasted compute budget, while a well-chosen one often improves final quality without any change to the model or the data.
Why the learning rate needs to change during training
The learning rate needs to change because the geometry of the loss landscape shifts as training progresses. Early on, parameters are far from a good solution and gradients are large and noisy, so the optimizer benefits from careful, small steps while its internal statistics settle. Later, the model moves through broad regions where larger steps make fast progress, and at the end it needs small steps to settle into a low-loss basin rather than bouncing around it.
In stochastic gradient descent and its variants, each update adds gradient noise whose magnitude scales with the learning rate. A high rate keeps the parameters exploring and acts as an implicit regularizer; a low rate reduces that noise and lets the loss fall further. Decaying the rate near the end therefore converts exploration into refinement, which is why the final drop in loss often coincides with the decay phase.
A schedule also protects adaptive optimizers such as Adam. Their second-moment estimates are unreliable in the first steps, so a full-size update at step one can push weights into unstable regions. A short warmup phase gives these estimates time to become meaningful before the optimizer takes full-size steps.
Common learning rate schedule shapes
Most schedules combine a warmup segment with a decay segment, and they differ mainly in the shape of the decay. The following table compares the shapes practitioners use most often.
| Schedule | Shape of the curve | Typical use |
|---|---|---|
| Constant | Flat value for the whole run | Quick experiments, small fine-tuning jobs |
| Step decay | Drops by a factor such as 10 at fixed epochs | Classic convolutional network training |
| Exponential decay | Multiplies by a factor slightly below 1 each step | Older vision and speech pipelines |
| Linear decay | Straight line from peak to zero or a floor | Transformer fine-tuning |
| Cosine decay | Half cosine from peak to a minimum | Pretraining of language and vision models |
| Warmup-stable-decay | Warmup, long plateau, short final decay | Large runs with flexible length |
| Inverse square root | Falls in proportion to one over the square root of the step | Original transformer recipes |
Cyclical schedules and cosine annealing with warm restarts periodically raise the rate again. They can help escape poor regions and produce several usable snapshots, but they are less common in large-scale pretraining, where a single monotone decay is easier to reason about.
Adaptive schedules such as reduce-on-plateau react to a monitored metric rather than following a fixed formula. They lower the rate by a set factor when validation loss stops improving for a chosen number of evaluations, which suits small projects but makes runs harder to reproduce and compare.
Key parameters of a schedule
A schedule is defined by a handful of parameters that interact with the optimizer, the batch size and the total number of steps. The ranges below are common starting points, not universal rules.
| Parameter | Common range | What it affects |
|---|---|---|
| Peak learning rate | 1e-4 to 3e-3 for Adam-type optimizers | Speed of progress and risk of divergence |
| Warmup length | About 0.5% to 5% of total steps | Stability of early updates |
| Minimum learning rate | 0 to 10% of the peak | How far the final loss can fall |
| Decay start | Immediately after warmup or after a plateau | Balance of exploration and refinement |
| Total schedule length | Matched to the planned token or step budget | Whether decay finishes when training ends |
The peak value usually scales with batch size: larger batches average out more gradient noise and tolerate larger steps, although the relationship is roughly linear only up to a critical batch size. Fine-tuning a pretrained model typically uses a peak one or two orders of magnitude lower than pretraining, because large steps can erase useful knowledge and contribute to catastrophic forgetting.
How a schedule relates to neighboring training concepts
A learning rate schedule is distinct from the learning rate itself and from optimizer mechanisms that also change effective step size. The table separates concepts that are frequently confused.
| Concept | What it controls | Relation to the schedule |
|---|---|---|
| Learning rate | Size of one update at a given moment | The value the schedule produces at each step |
| Adaptive optimizer scaling | Per-parameter step size from gradient statistics | Multiplies the scheduled global rate |
| Momentum | How past gradients influence the current update | Often scheduled inversely to the learning rate |
| Weight decay | Pull of weights toward zero | Coupled to the rate in many implementations |
| Gradient clipping | Maximum norm of a single update | Safety limit that complements warmup |
In decoupled weight decay, as in AdamW, the decay term is usually multiplied by the scheduled learning rate, so changing the schedule also changes the effective regularization strength. This coupling is a frequent source of confusion when practitioners compare runs that use different schedules.
Choosing a schedule in practice
The right schedule depends mostly on whether the training length is known in advance and on how much tuning budget is available. The decision table summarizes typical choices.
| Situation | Recommended choice | Reason |
|---|---|---|
| Fixed step budget, large pretraining | Warmup plus cosine decay | Well studied and robust across model sizes |
| Uncertain final length or continued pretraining | Warmup-stable-decay | Decay can start whenever training is stopped |
| Supervised fine-tuning of a pretrained model | Short warmup plus linear decay | Gentle updates preserve pretrained knowledge |
| Small model with limited tuning time | Reduce on plateau | Adapts automatically to validation loss |
| Exploring many snapshots for ensembling | Cosine with warm restarts | Each cycle ends at a usable model |
A practical workflow is to run a short learning rate range test, increasing the rate exponentially until the loss diverges, and to set the peak somewhat below the divergence point. Teams then fix the schedule shape and tune only the peak and warmup length, which keeps the search space small.
Failure modes and common mistakes
Most schedule problems show up as characteristic patterns in the loss curve, which makes them diagnosable without deep inspection of the model.
- Missing or too short warmup: early loss spikes or immediate divergence, especially with large batches and Adam-type optimizers.
- Schedule length mismatch: if the cosine schedule is configured for more steps than the run actually takes, the rate never decays and the final loss stays higher than it should.
- Restarting from a checkpoint with a reset schedule: the rate jumps back to its peak and can undo progress; the scheduler state must be restored together with the optimizer state.
- Comparing runs with different lengths: under cosine decay, a shorter run decays earlier, so intermediate checkpoints are not comparable across lengths.
- Decaying to exactly zero too early: the last part of training contributes almost nothing, which wastes compute.
A further misconception is that the schedule matters only for very large models. In practice, even small networks gain measurably from warmup and a final decay, and the effect is often larger than that of minor architecture changes.
Put simply
A learning rate schedule is a plan for how big each learning step should be at different moments of training. The model starts cautiously, then takes bigger steps once it knows roughly where it is going, and finally slows down to fine-tune the details.
It is similar to parking a car. You start slowly, drive at normal speed, and brake gently as you reach the space. Driving at full speed the whole time would overshoot, and crawling the whole time would take far too long.
In practice
A research team pretrains a language model with 1 billion parameters on a fixed budget of 100,000 optimizer steps using AdamW. They set a peak learning rate of 3e-4, a linear warmup over the first 2,000 steps and a cosine decay to 3e-5 at step 100,000. Gradient clipping at a norm of 1.0 provides an additional safeguard.
During an earlier trial without warmup, the loss spiked within the first 300 steps and training diverged. With the warmup in place, the loss falls smoothly, and the cosine decay produces a visible additional drop in validation loss over the final 15,000 steps. When the budget is later extended, the team switches to a warmup-stable-decay schedule so that the run can be stopped and decayed at any point without restarting from scratch.