A loss spike is a sudden, sharp increase in the training loss of a neural network that interrupts an otherwise smooth downward curve, often by a factor of two or more within a few optimizer steps. In artificial intelligence, and especially in Training of large language models, loss spikes are a symptom of optimization instability: the model briefly moves to a much worse region of its parameter space and either recovers on its own or diverges permanently.
Loss spikes matter because a single unrecovered spike can waste days of compute on a large training run. Practitioners therefore treat spike monitoring, prevention and recovery as a core part of training reliability, alongside checkpointing and careful choice of the learning rate schedule.
What happens inside the model during a loss spike
A loss spike occurs when one or a few parameter updates are far too large for the local curvature of the loss surface. Gradient descent assumes that a small step in the direction of the negative gradient lowers the loss; when the step overshoots a steep region, the loss rises instead, and the larger gradients that follow can amplify the error in a feedback loop.
In transformer models the instability often begins in specific components. Attention logits can grow very large, which pushes the softmax into a nearly one-hot state with tiny gradients; output logits can drift upward until the cross-entropy loss becomes numerically unstable; and the second-moment estimates of the Adam optimizer can lag behind a sudden increase in gradient size, so the effective step becomes much larger than intended.
The table below summarizes the most common internal mechanisms behind a spike.
| Mechanism | What goes wrong | Typical signal |
|---|---|---|
| Learning rate too high | Steps overshoot sharp regions of the loss surface | Spikes cluster near the end of warmup |
| Attention logit growth | Softmax saturates and gradients collapse | Rising maximum attention logit |
| Output logit drift | Logits grow until loss values become unstable | Large logit norm before the spike |
| Adam second-moment lag | Variance estimate is stale after rare large gradients | Sudden jump in update-to-weight ratio |
| Bad data batch | Corrupted, repetitive or extreme samples | Spike tied to a specific data shard |
| Numerical overflow | Values exceed the range of low-precision formats | Infinite or NaN values in activations |
Types of loss spikes and how they evolve
Not every loss spike is equally dangerous, and the shape of the curve after the spike tells practitioners how serious the event is. A transient spike returns to the previous trend within tens or hundreds of steps, while a divergent spike never recovers and usually ends with NaN values.
The following table compares the main spike patterns observed in large-scale training.
| Pattern | Behavior after the spike | Severity |
|---|---|---|
| Transient spike | Loss recovers to the prior trend within a few hundred steps | Low, usually tolerated |
| Slow-recovery spike | Loss recovers but stays above the trend for thousands of steps | Medium, wastes compute |
| Plateau after spike | Loss stops decreasing at a higher level | High, model quality degrades |
| Divergence | Loss keeps rising and turns into NaN | Critical, the run must be restarted |
| Recurring spikes | Spikes repeat at irregular intervals | High, indicates a systemic cause |
Spikes become more frequent as models grow larger and as learning rates are pushed higher to save compute. Smaller models trained with conservative settings may never show a spike, which is why problems sometimes appear only when a recipe is scaled up.
How loss spikes differ from related training problems
A loss spike is a short, abrupt event in the training loss, which distinguishes it from slower failure modes that share some symptoms. Correct diagnosis matters because each problem calls for a different remedy.
| Problem | Time scale | Main indicator | Typical remedy |
|---|---|---|---|
| Loss spike | A few to hundreds of steps | Sudden jump in training loss | Rewind, skip data, lower learning rate |
| Exploding gradients | Single steps | Very large gradient norm | Gradient clipping |
| Overfitting | Many epochs | Validation loss rises while training loss falls | Regularization, early stopping |
| Catastrophic forgetting | A fine-tuning phase | Old task performance drops | Data replay, lower learning rate |
| Underfitting | Entire run | Loss stays high throughout | Larger model, longer training |
Exploding gradients are often the immediate cause of a spike, but a spike can also occur with gradient norms that look normal when the optimizer state is miscalibrated. Overfitting, by contrast, never appears as a sudden jump in the training loss itself.
Preventing and handling loss spikes
Teams combine architectural safeguards, optimizer settings and operational procedures to keep spikes rare and recoverable. No single technique eliminates them, so the safest approach layers several defenses.
- Gradient clipping with a global norm threshold, often 1.0, limits the size of any single update.
- Learning rate warmup over the first thousands of steps lets the optimizer state stabilize before full-size updates.
- QK normalization and similar layer normalization variants keep attention logits within a safe range.
- Auxiliary z-loss penalizes large output logits and reduces logit drift.
- A lower Adam epsilon or adjusted beta2 value can reduce the lag of second-moment estimates.
- Mixed-precision training with bfloat16 avoids the narrow range of float16 that makes overflow likely.
When a serious spike occurs anyway, the standard response is a rewind-and-skip procedure: restore the last healthy checkpoint, skip the data batches that preceded the spike, and resume, sometimes with a temporarily reduced learning rate. The table below links common situations to the usual response.
| Situation | Recommended action | Reason |
|---|---|---|
| Single spike, quick recovery | Log it and continue | Restarting costs more than the brief loss |
| Spike tied to one data shard | Rewind and skip or clean that shard | The data is the likely trigger |
| Divergence to NaN | Rewind to the last checkpoint and lower the learning rate | The run cannot recover on its own |
| Recurring spikes | Add z-loss or QK normalization and adjust Adam settings | The cause is systemic, not random |
Common misconceptions about loss spikes
Several widespread assumptions lead to wrong diagnoses and wasted restarts. The most common ones are listed below.
- A spike always means bad data. Data can trigger a spike, but replaying the same batch from a different checkpoint often causes no spike, which shows that the optimizer state matters as much as the data.
- Gradient clipping prevents all spikes. Clipping caps the gradient norm, yet the Adam update can still be large when second-moment estimates are stale.
- Small spikes are harmless. A recovered spike can leave the model in a slightly worse basin, so frequent minor spikes may lower final quality.
- Spikes are random bad luck. Recurring spikes usually point to a systemic configuration problem that should be fixed rather than tolerated.
Put simply
Training an AI model is like slowly walking down a hill in fog, taking many small steps toward the lowest point. The loss is a number that shows how high up the model still is. Normally that number drops steadily.
A loss spike is a sudden jump back up the hill. The model takes a step that is too big, or trips over a strange piece of data, and ends up much worse for a moment. Often it finds its way back down quickly. Sometimes it falls off a cliff and cannot recover, and engineers must go back to a saved point and try again more carefully.
In practice
A research team pretrains a 7-billion-parameter language model on 1,000 GPUs, saving a checkpoint every 1,000 steps. At step 48,200 the training loss jumps from 2.1 to 4.8 within 30 steps, and the gradient norm rises sharply despite clipping at 1.0. After 400 steps the loss is still near 3.5 and climbing.
The team rewinds to the checkpoint at step 48,000, skips the next 300 data batches and lowers the peak learning rate by 20 percent for 2,000 steps. Inspection of the skipped batches reveals a shard of repetitive machine-generated text. The resumed run passes the same region smoothly, and the team loses only about four hours of compute instead of restarting from scratch. They also add an auxiliary z-loss for the rest of training to reduce the chance of another divergence.