Underfitting

On this page 7

Underfitting is the condition in which a machine learning model is too simple or too constrained to capture the real structure of its data, so it performs poorly on the training set and on unseen data alike. In model evaluation, underfitting is recognized by high training error that stays close to validation error, a signature of high bias rather than high variance.

Underfitting is the counterpart of overfitting. Where an overfitted model memorizes noise, an underfitted model misses genuine signal, and both conditions reduce the ability of artificial intelligence systems to generalize.

Why underfitting happens

Underfitting happens when the hypothesis space of the model cannot represent the target function or when training stops before the model reaches what it could represent. In both cases, the systematic error, or bias, dominates the total error.

The bias-variance decomposition explains the mechanism. Expected prediction error splits into bias squared, variance and irreducible noise. An underfitted model has low variance but high bias: retraining it on a different sample gives similar predictions, yet those predictions are consistently wrong in the same direction. A straight line fitted to a curved relationship is the classic illustration.

The causes fall into a small number of recurring groups, summarized below.

CauseMechanismTypical example
Insufficient model capacityToo few parameters or layersLinear model for a nonlinear boundary
Excessive regularizationPenalty shrinks weights too farVery large L2 coefficient or dropout rate
Uninformative featuresInputs lack predictive signalMissing interaction or domain features
Too little trainingOptimization stops earlyFew epochs or aggressive early stopping
Poor optimization settingsGradient descent stallsLearning rate far too small or too large
Common causes of underfitting

How to detect underfitting in model evaluation

Underfitting is detected by comparing training and validation performance against a meaningful baseline. If both errors are high and similar to each other, the model has not learned enough. If training error is low but validation error is high, the problem is overfitting instead.

Learning curves are the most informative diagnostic. They plot error against training set size or training time. In an underfitted model, both curves plateau early at a high error level and stay close together, which means that adding more data alone will not help. Cross-validation strengthens the diagnosis by showing that the high error is stable across folds rather than an artifact of one split.

The following table contrasts the diagnostic patterns of the three main regimes.

RegimeTraining errorValidation errorGap
UnderfittingHighHighSmall
Good fitLowLow, slightly higherSmall
OverfittingVery lowHighLarge
Diagnostic patterns of fit quality

Baselines matter because “high” error is relative. A classifier with 80 percent accuracy can be underfitted if a simple gradient-boosted model reaches 92 percent, or excellent if human experts agree only 82 percent of the time. Bayes error or human-level performance gives a realistic ceiling against which to judge bias.

  • Residual plots with visible structure, such as curves or funnels, indicate unexplained systematic patterns.
  • Uniform predictions, for example a classifier predicting the majority class almost always, suggest the model ignores its inputs.
  • Flat loss curves from the first epochs point to optimization problems rather than capacity limits.

Underfitting compared with overfitting

Underfitting and overfitting sit at opposite ends of the model complexity spectrum, and the goal of model selection is the region between them where validation error is lowest. Remedies for one condition usually worsen the other, which is why diagnosis must precede treatment.

AspectUnderfittingOverfitting
Dominant errorBiasVariance
Model complexityToo lowToo high
Effect of more dataLittle improvementUsually helps
Effect of regularizationMakes it worseUsually helps
Main remedyIncrease capacity or featuresConstrain or simplify the model
Underfitting and overfitting side by side

Very large neural networks complicate this classical picture through double descent, in which test error can fall again after the model passes the point where it fits training data exactly. Even so, underfitting remains common in practice whenever models are undertrained, overly regularized or fed weak features.

How to fix an underfitted model

Fixing underfitting means reducing bias by giving the model more expressive power or more useful information, then confirming the improvement on validation data. Changes should be made one at a time so that their individual effects remain measurable.

SituationRecommended actionReason
Model too simpleAdd layers, units or tree depthExpands the representable functions
Heavy regularizationLower L1, L2 or dropout strengthLets weights fit real patterns
Weak featuresEngineer interactions and domain featuresSupplies missing signal
Training stopped earlyTrain more epochs, relax early stoppingAllows the optimizer to converge
Stalled optimizationTune learning rate and scheduleRestores effective gradient descent
Wrong model familySwitch to boosting or a neural networkMatches the data structure
Remedies for underfitting

Each remedy carries a risk of overshooting into overfitting. After increasing capacity, practitioners should watch the gap between training and validation curves and reintroduce moderate regularization if the gap widens. Hyperparameter search with cross-validation automates much of this balancing.

Common mistakes when diagnosing underfitting

The most common mistake is collecting more data to fix a bias problem. Additional examples reduce variance, but they rarely help a model that is structurally unable to represent the pattern. Learning curves that have already flattened show that more data is not the bottleneck.

  • Ignoring label noise: if labels are inconsistent, high error may reflect the data ceiling rather than underfitting.
  • Judging by accuracy alone: on imbalanced data, accuracy can look acceptable while precision and recall for the minority class collapse.
  • Skipping the baseline: without a reference model, a modest score cannot be classified as underfitting or as a realistic limit.
  • Leaving regularization defaults untouched: library defaults are tuned for general cases and can be far too strong for small models.

The idea in brief

Underfitting means a model has not learned enough from its examples. Imagine a student who prepares for a geometry exam by memorizing only one rule: every shape is a square. The student gets most questions wrong in practice tests and in the real exam, because the rule is too simple for the subject.

A model in this state makes the same kind of mistake everywhere. It does badly on the data it studied and on new data. The cure is usually to give it more flexibility or better information, not simply more of the same examples.

A practical scenario

A data scientist at a regional energy utility builds a model to forecast hourly electricity demand. The first version is a linear regression using only temperature. Its mean absolute percentage error is about 14 percent on training data and 15 percent on validation data, and the residuals show a clear daily wave and a jump on weekends.

The small gap and structured residuals point to underfitting. The data scientist adds hour-of-day, day-of-week and holiday features, includes a squared temperature term to capture heating and cooling effects, and then switches to a gradient-boosted tree model with moderate depth.

The revised model reaches roughly 4 percent error on training data and 5 percent on validation data. The gap remains small, confirming that bias fell without introducing overfitting, and the operations team can now schedule generation reserves with far greater confidence.