Ground truth

On this page 7

Ground truth is the set of verified reference answers against which a model’s outputs are trained and judged: the correct label, value, boundary or response for each example in a dataset. Artificial intelligence systems built with supervised learning cannot learn or be evaluated without it, because every loss function and every accuracy metric compares a prediction with a value that is assumed to be correct.

The phrase comes from remote sensing, where satellite images were checked against measurements taken on the ground. In modern AI practice it rarely means absolute truth. It means the best available reference, usually produced by human annotators, instruments or records, and its quality sets an upper limit on measured performance.

How ground truth enters the learning loop

Ground truth plays two distinct roles: it is the target that training minimizes error against, and it is the yardstick that evaluation reports scores against. During training, the loss function compares each prediction with its reference label and produces a gradient; during evaluation, metrics such as accuracy, F1 score or mean absolute error summarize the same comparison on held-out data.

Because the reference is treated as correct, every error in it is learned or rewarded. A mislabeled training image pushes the weights in the wrong direction, and a mislabeled test image can make a correct prediction count as a mistake. For this reason practitioners separate training labels, which can tolerate some noise, from the evaluation set, which deserves the most careful verification.

The table below shows what ground truth looks like for common task types.

Task typeForm of ground truthTypical metric
Image classificationOne class label per imageAccuracy, top-5 accuracy
Object detectionBounding boxes with class labelsMean average precision
Semantic segmentationClass label per pixelIntersection over union
RegressionMeasured numeric valueMean absolute error, RMSE
Speech recognitionVerified transcriptWord error rate
Language generationOne or more reference answers or rubricsExact match, human preference
Ground truth formats by task type

Where ground truth comes from

Ground truth is produced by a labeling process, and the source determines both its cost and its reliability. The main options are expert annotation, crowdsourced annotation, instrument measurement, transactional records and outcomes observed later in time.

Each source carries its own characteristic weakness, summarized below.

SourceStrengthMain weakness
Domain expertsHigh accuracy on difficult casesSlow, expensive, small volume
Crowd annotatorsLarge volume at low costInconsistent quality, needs aggregation
Sensors and instrumentsObjective numeric valuesCalibration drift and measurement error
Business recordsAlready available at scaleReflect past decisions and their biases
Delayed outcomesClose to the real targetArrive weeks or years after prediction
Model-assisted labelsFast pre-annotationCan copy the model’s own errors
Sources of ground truth and their trade-offs

Mature teams combine sources: a model proposes labels, annotators correct them, and adjudication by a senior reviewer settles disagreements. Written annotation guidelines with edge-case examples matter as much as the people applying them, because ambiguous instructions produce labels that differ for reasons unrelated to the data.

Measuring the quality of ground truth

The quality of ground truth is measured by inter-annotator agreement, the degree to which independent labelers assign the same answer to the same item. Common statistics are Cohen’s kappa for two annotators, Fleiss’ kappa for several and Krippendorff’s alpha for mixed data types; all correct raw agreement for the agreement expected by chance.

Agreement levels set a realistic ceiling. If two radiologists agree on only 80 percent of cases, a model reported at 95 percent accuracy against one of them is probably fitting that annotator’s habits rather than the underlying condition. Studies of widely used benchmarks have found label error rates of a few percent in test sets, which is enough to reorder closely ranked models.

StatisticUse caseRough interpretation
Percent agreementQuick check, any number of ratersIgnores chance agreement
Cohen’s kappaTwo raters, categorical labelsAbove 0.8 is usually considered strong
Fleiss’ kappaThree or more ratersSame scale as Cohen’s kappa
Krippendorff’s alphaMissing ratings, ordinal or interval dataAbove 0.8 is a common target
Agreement statistics used to audit labels

Ground truth is not the same as training data, although the two are often discussed together. Training data is the full collection of examples fed to a model; ground truth is the reference portion of that collection, the answers attached to the inputs. Unsupervised learning uses training data with no ground truth at all, while self-supervised learning derives labels from the data itself, for example by hiding a word and asking the model to predict it.

ConceptWhat it refers toRelation to ground truth
LabelA single annotation on one exampleOne unit of ground truth
Training dataAll inputs used to fit a modelContains ground truth in supervised learning
Gold standardThe best accepted reference methodOften the source of ground truth
Pseudo-labelA label predicted by a modelA substitute, not verified ground truth
Silver standardLabels produced automatically at scaleCheaper but noisier ground truth
How ground truth differs from neighboring terms

Limitations and common mistakes

The most persistent misconception is that ground truth is objective by definition. In many tasks, such as toxicity rating, sentiment or medical triage, reasonable experts disagree, and forcing a single label hides genuine ambiguity that the model will later meet in production.

  • Label leakage: the reference is derived from information that will not exist at prediction time, which inflates offline scores.
  • Proxy targets: an arrest record stands in for crime, or a click stands in for satisfaction, and the model learns the proxy rather than the goal.
  • Stale references: definitions or real-world conditions change, but old labels remain, a form of concept drift in the reference itself.
  • Single-annotator test sets: evaluation depends on one person’s judgment, with no way to estimate label noise.
  • Benchmark contamination: test items and their answers appear in training data, so scores no longer measure generalization.

Remedies include multiple annotations per item, explicit uncertainty labels, periodic re-annotation and keeping a small, carefully adjudicated gold evaluation set that is never used for training. This data-centric view treats label quality as an engineering asset rather than a fixed input.

A plain-language summary

Ground truth is the answer key. When a computer system learns from examples, someone first has to say what the right answer is for each one: this photo shows a cat, this email is spam, this house sold for a certain price. The system practices against that answer key and is later graded with it.

The catch is that an answer key can contain mistakes. If the teacher’s key is wrong, a student who answers correctly gets marked down, and a student who copies the key learns the errors. That is why careful teams check their answer keys with several people before trusting the grades.

An example from practice

A hospital team builds a model that flags pneumonia on chest X-rays. Their first ground truth comes from the original radiology reports, which are extracted automatically from free text. The model scores 92 percent accuracy on a test set labeled the same way.

Before deployment, the team asks three radiologists to label 1,000 test images independently. Agreement between them reaches a Fleiss’ kappa of about 0.7, and about 6 percent of the report-derived labels turn out to be wrong. Measured against the adjudicated expert labels, the model’s accuracy falls to 86 percent, and most of its remaining errors fall on cases where the radiologists themselves disagreed.

The team rebuilds its evaluation set from adjudicated labels, adds an uncertain category and retrains. The outcome is a lower but trustworthy score and a model whose behavior on difficult images is now understood before it reaches clinicians.