Learning rate schedule

Also known as: learning rate scheduler · learning rate decay

On this page 8

A learning rate schedule is a rule that changes the learning rate over the course of training, typically as a function of the current step or epoch, instead of holding it at one fixed value. Training of modern artificial intelligence models, from image classifiers to large language models, almost always relies on a schedule, because the step size that works best at the start of optimization is rarely the one that works best near the end.

Also called a learning rate scheduler or, when it only reduces the value, learning rate decay, the schedule is one of the most influential hyperparameters in gradient-based optimization. A poorly chosen schedule can cause loss spikes, divergence or a wasted compute budget, while a well-chosen one often improves final quality without any change to the model or the data.

Why the learning rate needs to change during training

The learning rate needs to change because the geometry of the loss landscape shifts as training progresses. Early on, parameters are far from a good solution and gradients are large and noisy, so the optimizer benefits from careful, small steps while its internal statistics settle. Later, the model moves through broad regions where larger steps make fast progress, and at the end it needs small steps to settle into a low-loss basin rather than bouncing around it.

In stochastic gradient descent and its variants, each update adds gradient noise whose magnitude scales with the learning rate. A high rate keeps the parameters exploring and acts as an implicit regularizer; a low rate reduces that noise and lets the loss fall further. Decaying the rate near the end therefore converts exploration into refinement, which is why the final drop in loss often coincides with the decay phase.

A schedule also protects adaptive optimizers such as Adam. Their second-moment estimates are unreliable in the first steps, so a full-size update at step one can push weights into unstable regions. A short warmup phase gives these estimates time to become meaningful before the optimizer takes full-size steps.

Common learning rate schedule shapes

Most schedules combine a warmup segment with a decay segment, and they differ mainly in the shape of the decay. The following table compares the shapes practitioners use most often.

ScheduleShape of the curveTypical use
ConstantFlat value for the whole runQuick experiments, small fine-tuning jobs
Step decayDrops by a factor such as 10 at fixed epochsClassic convolutional network training
Exponential decayMultiplies by a factor slightly below 1 each stepOlder vision and speech pipelines
Linear decayStraight line from peak to zero or a floorTransformer fine-tuning
Cosine decayHalf cosine from peak to a minimumPretraining of language and vision models
Warmup-stable-decayWarmup, long plateau, short final decayLarge runs with flexible length
Inverse square rootFalls in proportion to one over the square root of the stepOriginal transformer recipes
Widely used schedule shapes and their behavior

Cyclical schedules and cosine annealing with warm restarts periodically raise the rate again. They can help escape poor regions and produce several usable snapshots, but they are less common in large-scale pretraining, where a single monotone decay is easier to reason about.

Adaptive schedules such as reduce-on-plateau react to a monitored metric rather than following a fixed formula. They lower the rate by a set factor when validation loss stops improving for a chosen number of evaluations, which suits small projects but makes runs harder to reproduce and compare.

Key parameters of a schedule

A schedule is defined by a handful of parameters that interact with the optimizer, the batch size and the total number of steps. The ranges below are common starting points, not universal rules.

ParameterCommon rangeWhat it affects
Peak learning rate1e-4 to 3e-3 for Adam-type optimizersSpeed of progress and risk of divergence
Warmup lengthAbout 0.5% to 5% of total stepsStability of early updates
Minimum learning rate0 to 10% of the peakHow far the final loss can fall
Decay startImmediately after warmup or after a plateauBalance of exploration and refinement
Total schedule lengthMatched to the planned token or step budgetWhether decay finishes when training ends
Schedule parameters and what they control

The peak value usually scales with batch size: larger batches average out more gradient noise and tolerate larger steps, although the relationship is roughly linear only up to a critical batch size. Fine-tuning a pretrained model typically uses a peak one or two orders of magnitude lower than pretraining, because large steps can erase useful knowledge and contribute to catastrophic forgetting.

How a schedule relates to neighboring training concepts

A learning rate schedule is distinct from the learning rate itself and from optimizer mechanisms that also change effective step size. The table separates concepts that are frequently confused.

ConceptWhat it controlsRelation to the schedule
Learning rateSize of one update at a given momentThe value the schedule produces at each step
Adaptive optimizer scalingPer-parameter step size from gradient statisticsMultiplies the scheduled global rate
MomentumHow past gradients influence the current updateOften scheduled inversely to the learning rate
Weight decayPull of weights toward zeroCoupled to the rate in many implementations
Gradient clippingMaximum norm of a single updateSafety limit that complements warmup
Learning rate schedule compared with related concepts

In decoupled weight decay, as in AdamW, the decay term is usually multiplied by the scheduled learning rate, so changing the schedule also changes the effective regularization strength. This coupling is a frequent source of confusion when practitioners compare runs that use different schedules.

Choosing a schedule in practice

The right schedule depends mostly on whether the training length is known in advance and on how much tuning budget is available. The decision table summarizes typical choices.

SituationRecommended choiceReason
Fixed step budget, large pretrainingWarmup plus cosine decayWell studied and robust across model sizes
Uncertain final length or continued pretrainingWarmup-stable-decayDecay can start whenever training is stopped
Supervised fine-tuning of a pretrained modelShort warmup plus linear decayGentle updates preserve pretrained knowledge
Small model with limited tuning timeReduce on plateauAdapts automatically to validation loss
Exploring many snapshots for ensemblingCosine with warm restartsEach cycle ends at a usable model
Situation and recommended schedule

A practical workflow is to run a short learning rate range test, increasing the rate exponentially until the loss diverges, and to set the peak somewhat below the divergence point. Teams then fix the schedule shape and tune only the peak and warmup length, which keeps the search space small.

Failure modes and common mistakes

Most schedule problems show up as characteristic patterns in the loss curve, which makes them diagnosable without deep inspection of the model.

  • Missing or too short warmup: early loss spikes or immediate divergence, especially with large batches and Adam-type optimizers.
  • Schedule length mismatch: if the cosine schedule is configured for more steps than the run actually takes, the rate never decays and the final loss stays higher than it should.
  • Restarting from a checkpoint with a reset schedule: the rate jumps back to its peak and can undo progress; the scheduler state must be restored together with the optimizer state.
  • Comparing runs with different lengths: under cosine decay, a shorter run decays earlier, so intermediate checkpoints are not comparable across lengths.
  • Decaying to exactly zero too early: the last part of training contributes almost nothing, which wastes compute.

A further misconception is that the schedule matters only for very large models. In practice, even small networks gain measurably from warmup and a final decay, and the effect is often larger than that of minor architecture changes.

Put simply

A learning rate schedule is a plan for how big each learning step should be at different moments of training. The model starts cautiously, then takes bigger steps once it knows roughly where it is going, and finally slows down to fine-tune the details.

It is similar to parking a car. You start slowly, drive at normal speed, and brake gently as you reach the space. Driving at full speed the whole time would overshoot, and crawling the whole time would take far too long.

In practice

A research team pretrains a language model with 1 billion parameters on a fixed budget of 100,000 optimizer steps using AdamW. They set a peak learning rate of 3e-4, a linear warmup over the first 2,000 steps and a cosine decay to 3e-5 at step 100,000. Gradient clipping at a norm of 1.0 provides an additional safeguard.

During an earlier trial without warmup, the loss spiked within the first 300 steps and training diverged. With the warmup in place, the loss falls smoothly, and the cosine decay produces a visible additional drop in validation loss over the final 15,000 steps. When the budget is later extended, the team switches to a warmup-stable-decay schedule so that the run can be stopped and decayed at any point without restarting from scratch.