Early stopping

On this page 7

Early stopping is a regularization technique in model training that halts optimization when performance on held-out validation data stops improving, and then keeps the parameters from the best observed checkpoint. In artificial intelligence and machine learning, it is one of the oldest and cheapest defenses against overfitting, because it treats the number of training steps itself as a hyperparameter chosen from evidence rather than fixed in advance.

The method rests on a simple observation: as training continues, training loss keeps falling while validation loss usually falls, flattens and eventually rises. The point where the validation curve turns upward marks the moment the model starts memorizing noise instead of learning patterns, and early stopping aims to end training close to that point.

How early stopping monitors the validation curve

Early stopping works by evaluating a monitored metric at regular intervals and comparing each new value with the best value seen so far. The monitored quantity is typically validation loss, although accuracy, F1 score, perplexity or a task-specific score can serve the same purpose when it reflects the real objective better.

A practical implementation needs a small set of rules. The patience parameter defines how many evaluations without improvement are tolerated before training ends, and a minimum improvement threshold, often called min_delta, prevents tiny random fluctuations from counting as progress. When the rule fires, the procedure restores the best checkpoint rather than the last one, because the last weights already reflect the degraded epochs.

The table below lists the main settings of an early stopping rule and how each one shapes the outcome.

ParameterTypical valuesEffect on training
Monitored metricValidation loss, accuracy, F1Defines what counts as improvement
Patience3-20 evaluationsHigher values tolerate noisy plateaus
Minimum delta0 to about 0.1% of the metricIgnores negligible gains
Evaluation intervalEvery epoch or every N stepsSets the granularity of the decision
Restore best weightsEnabled or disabledDetermines which checkpoint is kept
Warm-up periodFirst 1-5 epochs or a step countAvoids stopping during unstable early phases
Core parameters of an early stopping rule

Why stopping early acts as regularization

Stopping early limits the effective capacity of a model, because parameters that start near small initial values cannot travel far from them within a limited number of updates. For linear models trained with gradient descent and a quadratic loss, theory shows that early stopping behaves approximately like L2 regularization, where the number of steps multiplied by the learning rate plays the role of an inverse weight decay coefficient.

This equivalence is only approximate for deep networks, but the intuition holds. Early training captures broad, high-signal structure in the data, while later training increasingly fits rare examples, label errors and idiosyncratic features. Ending training before that second phase dominates trades a slightly higher training loss for better generalization to unseen data.

The following comparison places early stopping beside other regularization methods used in the same training pipelines.

MethodWhat it constrainsExtra costMain drawback
Early stoppingNumber of optimization stepsValidation passesDepends on a reliable validation set
Weight decayMagnitude of the weightsNegligibleRequires tuning a coefficient
DropoutCo-adaptation of unitsSlower convergenceLess common in large language models
Data augmentationSensitivity to input variationMore data processingNeeds domain-valid transformations
Label smoothingOverconfident predictionsNegligibleCan hurt calibration in some settings
Early stopping compared with related regularization methods

Variants of the stopping criterion

Several stopping criteria exist because validation curves are noisy and rarely have one clean minimum. The simplest rule stops at the first increase, which is fragile; practical systems use patience, smoothing or relative thresholds to separate genuine deterioration from evaluation noise.

  • Patience-based stopping ends training after a fixed number of evaluations without a new best value.
  • Generalization loss criteria stop when validation loss exceeds its best value by a set percentage, such as 5%.
  • Smoothed criteria apply a moving average to the metric before comparing values.
  • Budget-aware stopping combines a patience rule with a hard cap on steps or compute.

Choosing among these variants depends on how noisy the metric is and how expensive each evaluation is. The decision table below summarizes common situations.

SituationRecommended setupReason
Small tabular datasetPatience of 10-20 with best weights restoredValidation noise is high
Large image classifierPatience of 5-10 epochsCurves are smoother on large sets
Fine-tuning a pretrained modelEvaluate every few hundred steps, patience 3-5Overfitting appears within one or two epochs
Gradient-boosted treesStop after 50-200 rounds without gainEach round is cheap to evaluate
Large-scale pretrainingFixed compute budget insteadValidation loss rarely rises in one pass
Choosing an early stopping setup

Early stopping in fine-tuning and large models

Early stopping matters most when data is scarce relative to model size, which is exactly the case in fine-tuning. When a pretrained language model is adapted with supervised fine-tuning, LoRA or QLoRA on a few thousand examples, validation loss often reaches its minimum within one to three epochs, and further training degrades general abilities and can accelerate catastrophic forgetting.

Large-scale pretraining behaves differently. Models trained on trillions of tokens usually see each example once or a few times, so validation loss keeps decreasing for the entire run and the stopping point is set by a compute budget derived from neural scaling laws. In that regime, practitioners monitor validation curves mainly to detect loss spikes and data problems, not to choose when to stop.

The interaction with the learning rate schedule also deserves attention. With cosine or linear decay, the lowest validation loss often appears near the end of the schedule, so an aggressive patience setting can stop a run before the learning rate has annealed and leave quality on the table.

Limitations and common mistakes

Early stopping is simple, but several mistakes undermine it. The most serious is selecting the checkpoint on the test set, which leaks test information into model selection and inflates reported results; the stopping decision must use a separate validation set.

  • Too little patience stops training on a temporary plateau, especially with noisy metrics or small validation sets.
  • Monitoring the wrong metric can stop on loss while the task metric, such as accuracy, is still improving.
  • Forgetting to restore the best weights silently ships a model from several degraded epochs later.
  • Ignoring double descent can end training just before a second improvement phase in heavily overparameterized models.

A further subtlety is that repeated tuning against the same validation set gradually overfits the validation data itself. When many runs and stopping rules are compared, cross-validation or a final untouched holdout set gives a more honest estimate of generalization.

In plain words

Training a model is a bit like a student preparing for an exam with a set of practice questions. At first, the student learns the real ideas behind the questions. If the student keeps going for too long, they start memorizing the exact answers instead, which does not help on new questions.

Early stopping is the teacher who regularly gives a short quiz with questions the student has never seen. As long as quiz scores improve, studying continues. When the scores stop improving for a while, the teacher says stop here and keep the best version. The result is a model that performs well on new data, not only on the examples it practiced with.

A real-world example

A support team at a mid-sized software company fine-tunes an open-weight language model to classify incoming tickets into 40 categories. The team has about 12,000 labeled tickets, sets aside 1,500 of them as a validation set and trains with a learning rate of 2e-5 for a planned maximum of five epochs.

The training script evaluates validation loss and macro F1 score every 250 steps, with a patience of four evaluations and best-weight restoration enabled. Training loss falls steadily for the whole run, but validation F1 peaks at 0.87 midway through the second epoch and then slowly declines as the model begins to memorize rare ticket phrasings.

The rule halts training early in the third epoch and restores the checkpoint from the peak. The team saves roughly 40% of the planned GPU time, and the restored model holds an F1 score of 0.86 on a separate test set, confirming that the selected checkpoint generalizes rather than reflecting a lucky validation result.