Loss spike

On this page 7

A loss spike is a sudden, sharp increase in the training loss of a neural network that interrupts an otherwise smooth downward curve, often by a factor of two or more within a few optimizer steps. In artificial intelligence, and especially in Training of large language models, loss spikes are a symptom of optimization instability: the model briefly moves to a much worse region of its parameter space and either recovers on its own or diverges permanently.

Loss spikes matter because a single unrecovered spike can waste days of compute on a large training run. Practitioners therefore treat spike monitoring, prevention and recovery as a core part of training reliability, alongside checkpointing and careful choice of the learning rate schedule.

What happens inside the model during a loss spike

A loss spike occurs when one or a few parameter updates are far too large for the local curvature of the loss surface. Gradient descent assumes that a small step in the direction of the negative gradient lowers the loss; when the step overshoots a steep region, the loss rises instead, and the larger gradients that follow can amplify the error in a feedback loop.

In transformer models the instability often begins in specific components. Attention logits can grow very large, which pushes the softmax into a nearly one-hot state with tiny gradients; output logits can drift upward until the cross-entropy loss becomes numerically unstable; and the second-moment estimates of the Adam optimizer can lag behind a sudden increase in gradient size, so the effective step becomes much larger than intended.

The table below summarizes the most common internal mechanisms behind a spike.

MechanismWhat goes wrongTypical signal
Learning rate too highSteps overshoot sharp regions of the loss surfaceSpikes cluster near the end of warmup
Attention logit growthSoftmax saturates and gradients collapseRising maximum attention logit
Output logit driftLogits grow until loss values become unstableLarge logit norm before the spike
Adam second-moment lagVariance estimate is stale after rare large gradientsSudden jump in update-to-weight ratio
Bad data batchCorrupted, repetitive or extreme samplesSpike tied to a specific data shard
Numerical overflowValues exceed the range of low-precision formatsInfinite or NaN values in activations
Typical mechanisms behind loss spikes

Types of loss spikes and how they evolve

Not every loss spike is equally dangerous, and the shape of the curve after the spike tells practitioners how serious the event is. A transient spike returns to the previous trend within tens or hundreds of steps, while a divergent spike never recovers and usually ends with NaN values.

The following table compares the main spike patterns observed in large-scale training.

PatternBehavior after the spikeSeverity
Transient spikeLoss recovers to the prior trend within a few hundred stepsLow, usually tolerated
Slow-recovery spikeLoss recovers but stays above the trend for thousands of stepsMedium, wastes compute
Plateau after spikeLoss stops decreasing at a higher levelHigh, model quality degrades
DivergenceLoss keeps rising and turns into NaNCritical, the run must be restarted
Recurring spikesSpikes repeat at irregular intervalsHigh, indicates a systemic cause
Spike patterns and their usual severity

Spikes become more frequent as models grow larger and as learning rates are pushed higher to save compute. Smaller models trained with conservative settings may never show a spike, which is why problems sometimes appear only when a recipe is scaled up.

A loss spike is a short, abrupt event in the training loss, which distinguishes it from slower failure modes that share some symptoms. Correct diagnosis matters because each problem calls for a different remedy.

ProblemTime scaleMain indicatorTypical remedy
Loss spikeA few to hundreds of stepsSudden jump in training lossRewind, skip data, lower learning rate
Exploding gradientsSingle stepsVery large gradient normGradient clipping
OverfittingMany epochsValidation loss rises while training loss fallsRegularization, early stopping
Catastrophic forgettingA fine-tuning phaseOld task performance dropsData replay, lower learning rate
UnderfittingEntire runLoss stays high throughoutLarger model, longer training
Loss spikes compared with neighboring problems

Exploding gradients are often the immediate cause of a spike, but a spike can also occur with gradient norms that look normal when the optimizer state is miscalibrated. Overfitting, by contrast, never appears as a sudden jump in the training loss itself.

Preventing and handling loss spikes

Teams combine architectural safeguards, optimizer settings and operational procedures to keep spikes rare and recoverable. No single technique eliminates them, so the safest approach layers several defenses.

  • Gradient clipping with a global norm threshold, often 1.0, limits the size of any single update.
  • Learning rate warmup over the first thousands of steps lets the optimizer state stabilize before full-size updates.
  • QK normalization and similar layer normalization variants keep attention logits within a safe range.
  • Auxiliary z-loss penalizes large output logits and reduces logit drift.
  • A lower Adam epsilon or adjusted beta2 value can reduce the lag of second-moment estimates.
  • Mixed-precision training with bfloat16 avoids the narrow range of float16 that makes overflow likely.

When a serious spike occurs anyway, the standard response is a rewind-and-skip procedure: restore the last healthy checkpoint, skip the data batches that preceded the spike, and resume, sometimes with a temporarily reduced learning rate. The table below links common situations to the usual response.

SituationRecommended actionReason
Single spike, quick recoveryLog it and continueRestarting costs more than the brief loss
Spike tied to one data shardRewind and skip or clean that shardThe data is the likely trigger
Divergence to NaNRewind to the last checkpoint and lower the learning rateThe run cannot recover on its own
Recurring spikesAdd z-loss or QK normalization and adjust Adam settingsThe cause is systemic, not random
Decision guide for responding to a spike

Common misconceptions about loss spikes

Several widespread assumptions lead to wrong diagnoses and wasted restarts. The most common ones are listed below.

  • A spike always means bad data. Data can trigger a spike, but replaying the same batch from a different checkpoint often causes no spike, which shows that the optimizer state matters as much as the data.
  • Gradient clipping prevents all spikes. Clipping caps the gradient norm, yet the Adam update can still be large when second-moment estimates are stale.
  • Small spikes are harmless. A recovered spike can leave the model in a slightly worse basin, so frequent minor spikes may lower final quality.
  • Spikes are random bad luck. Recurring spikes usually point to a systemic configuration problem that should be fixed rather than tolerated.

Put simply

Training an AI model is like slowly walking down a hill in fog, taking many small steps toward the lowest point. The loss is a number that shows how high up the model still is. Normally that number drops steadily.

A loss spike is a sudden jump back up the hill. The model takes a step that is too big, or trips over a strange piece of data, and ends up much worse for a moment. Often it finds its way back down quickly. Sometimes it falls off a cliff and cannot recover, and engineers must go back to a saved point and try again more carefully.

In practice

A research team pretrains a 7-billion-parameter language model on 1,000 GPUs, saving a checkpoint every 1,000 steps. At step 48,200 the training loss jumps from 2.1 to 4.8 within 30 steps, and the gradient norm rises sharply despite clipping at 1.0. After 400 steps the loss is still near 3.5 and climbing.

The team rewinds to the checkpoint at step 48,000, skips the next 300 data batches and lowers the peak learning rate by 20 percent for 2,000 steps. Inspection of the skipped batches reveals a shard of repetitive machine-generated text. The resumed run passes the same region smoothly, and the team loses only about four hours of compute instead of restarting from scratch. They also add an auxiliary z-loss for the rest of training to reduce the chance of another divergence.