Validation set

Also known as: dev set

On this page 7

A validation set, also called a dev set, is a portion of labeled data that is held out from training and used during development to tune hyperparameters, compare candidate models and decide when to stop training. In model evaluation for artificial intelligence, it sits between the training set, which the model learns from, and the test set, which is reserved for the final, unbiased estimate of performance.

The validation set exists because a model cannot be judged fairly on the data it learned from. Yet every decision based on validation scores leaks a little information into the model, so the validation set gradually becomes optimistic and cannot replace a truly untouched test set.

How a validation set fits into the data split

A validation set is one of three disjoint partitions of the available labeled data, each with a distinct job. The training set fits the model parameters, the validation set guides design decisions, and the test set measures the final generalization once all decisions are frozen.

PartitionPurposeUsed byHow often it is consulted
Training setFit model weightsOptimizerEvery step
Validation setTune and select modelsPractitioner and automated searchMany times during development
Test setReport final performanceFinal evaluation onlyIdeally once
Roles of the three data partitions

The key distinction is who is allowed to react to the score. The optimizer never sees validation labels, but the engineer and any hyperparameter search do. That indirect feedback is what separates the validation set from the test set, even when both are drawn from the same distribution.

What the validation set is used for

The validation set supports every development decision that depends on measured performance rather than on training loss. Typical uses include the following.

  • Hyperparameter tuning: choosing the learning rate, regularization strength, model size or number of trees.
  • Early stopping: halting training when validation loss stops improving, which limits overfitting.
  • Model selection: comparing architectures or feature sets and keeping the strongest candidate.
  • Threshold calibration: setting a decision threshold that balances precision and recall for the target use.
  • Error analysis: inspecting misclassified validation examples to guide the next iteration.

For large language models, the validation set often takes the form of a held-out loss or perplexity on unseen text, plus small task suites checked at regular checkpoints. These signals detect divergence, data mixture problems and overfitting long before a full benchmark run.

Sizing and building a reliable validation set

A good validation set is large enough to separate real improvements from noise and representative of the data the model will meet in use. Size and composition matter more than the exact percentage of the dataset.

The statistical rule is simple: the smaller the difference you want to detect, the more examples you need. With 1,000 examples, the standard error of an accuracy near 90 percent is about one percentage point, so a 0.5-point gain is indistinguishable from chance. The table below gives common sizing practice.

Total labeled dataCommon validation sharePractical note
Under 10,000 examples15 to 20 percentPrefer cross-validation instead of one split
10,000 to 1 million5 to 15 percentA few thousand examples usually suffice
Over 1 million1 percent or lessAbsolute count matters more than share
Rare positive classEnough positives for stable recallUse stratified sampling
Typical validation set sizes by dataset scale

Composition rules are equally important. Split by user, patient, document or time whenever examples are correlated, so near-duplicates do not appear on both sides. For forecasting, the validation period must follow the training period in time, never be sampled randomly from it.

The single held-out validation set is the simplest of several validation schemes, and the right choice depends on data volume, compute budget and how correlated the examples are.

ApproachHow it worksStrengthWeakness
Hold-out validation setOne fixed splitCheap and simpleHigh variance on small data
K-fold cross-validationRotate k validation foldsUses all data for both rolesk times the training cost
Nested cross-validationInner loop tunes, outer loop evaluatesUnbiased tuning estimateVery expensive
Time-based validationValidate on later periodsRespects temporal orderFewer usable periods
Benchmark evaluationPublic fixed test suiteComparable across teamsProne to benchmark contamination
Validation approaches and their trade-offs

Cross-validation is preferred for small tabular datasets, while deep learning projects usually rely on one fixed validation set because retraining a large network k times is too costly.

Common mistakes with validation sets

Most validation failures produce scores that look better than real-world performance. The table below lists frequent problems, how they show up and how to correct them.

MistakeSymptomFix
Data leakage between splitsValidation far above production resultsDeduplicate and split by entity
Overfitting to the validation setValidation gains do not transfer to testRefresh validation data or use a second holdout
Using the test set for tuningOptimistic published numbersLock the test set until the end
Preprocessing fitted on all dataSmall but systematic inflationFit scalers and vocabularies on training only
Distribution shift ignoredGood validation, poor live accuracyBuild validation from recent, realistic data
Validation set problems, symptoms and fixes

The subtlest issue is adaptive overfitting: after hundreds of experiments judged on the same validation set, the winning configuration partly fits that sample’s noise. Teams that iterate heavily should rotate or enlarge the validation data and confirm results on the test set only at milestones.

The idea in brief

When a model learns from examples, it needs a fair way to check its progress. A validation set is a practice exam made of questions the model has never studied. The builders use it to try different settings and keep the version that scores best.

There is a catch. If you keep adjusting your study plan to the same practice exam, you start to learn that exam instead of the subject. That is why a separate final exam, the test set, stays locked away until the very end and gives the honest final grade.

A practical scenario

A hospital analytics team builds a model that flags chest X-rays for urgent review using 60,000 labeled images from 18,000 patients. They split the data by patient, not by image, into 70 percent training, 15 percent validation and 15 percent test, so no patient appears in two partitions.

Over three weeks they run about 40 experiments with different learning rates, image resolutions and augmentation settings, judging each on the validation set’s recall at 95 percent specificity. Early stopping halts most runs after 12 epochs, when validation loss begins to rise. The best configuration reaches 0.88 recall on validation and 0.86 on the untouched test set, a small gap that confirms the validation set guided the team without becoming badly overfitted.