Representation bias

On this page 7

Representation bias is a systematic error that arises when the data used to build an AI system does not adequately reflect the population on which the system will be used, because some groups, conditions or contexts are missing, under-sampled or recorded differently. In AI ethics, it is one of the most common root causes of unfair outcomes, since a model learns less reliable patterns for the people it rarely sees.

The problem sits in the data collection and sampling stage, before any model is trained. It can exist even when every individual record is accurate and the learning algorithm is neutral, which is why it is often invisible in aggregate accuracy figures and only appears when performance is broken down by subgroup.

How representation bias enters a dataset

Representation bias occurs when the sampled population differs from the target population in ways that matter for the task. The gap usually comes from convenience: data is gathered where it is cheap, plentiful or already digitized, not where it is representative.

The most frequent sources are summarized below, together with a typical illustration of each.

SourceMechanismTypical illustration
Sampling frame gapsSome groups never enter the poolHealth data from insured hospital patients only
Under-samplingGroups appear, but in very small numbersFew darker skin tones in a dermatology image set
Geographic skewData concentrated in a few regionsStreet images mostly from North America and Europe
Temporal mismatchData reflects an older populationCredit records collected before a market change
Platform skewData from users of one service or deviceSpeech samples recorded on high-end phones
Participation biasCertain people opt out or are excludedSurveys answered mainly by younger, online users
Common sources of representation bias

Because a model minimizes average error across the training set, it allocates most of its capacity to the majority patterns. Minority patterns contribute little to the loss, so errors on those groups are cheap for the optimizer and costly for the people affected.

Representation bias is one of several data-related biases described in fairness research, and distinguishing them matters because each requires a different remedy. The comparison below places it next to its closest neighbors in AI ethics.

ConceptCore problemCan more data fix it
Representation biasWho is in the data, and how manyYes, if collected from missing groups
Historical biasThe world itself reflects past inequityNo, accurate data repeats the inequity
Measurement biasFeatures or labels measured unequallyNo, the measurement must change
Bias amplificationModel exaggerates skews in the dataPartly, with balanced data and constraints
Representational harmOutputs demean or stereotype a groupSometimes, if caused by skewed data
Representation bias compared with neighboring concepts

The key distinction is with historical bias. If a dataset perfectly mirrors a society in which one group was historically denied loans, the data is representative but still encodes injustice. Representation bias, by contrast, is a mismatch between sample and population, and it can be reduced by better sampling.

How to detect representation bias

Detection starts with comparing the dataset composition to the deployment population, then checking whether model performance differs across subgroups. Neither step is possible without recording, or at least estimating, the relevant attributes.

  • Composition audit: count records per group and compare the shares with census, customer or clinical reference figures.
  • Disaggregated evaluation: report accuracy, false positive rate and false negative rate for each subgroup separately, not only overall.
  • Intersectional slices: test combinations such as age and gender together, because small intersections are often the most under-represented.
  • Documentation: record collection methods and known gaps in datasheets for datasets so later users understand the limits.

Subgroup metrics need enough samples to be meaningful. With only a few dozen test cases for a group, confidence intervals become wide, so an apparently fair result may simply reflect too little evidence to detect a gap.

Mitigation strategies and their trade-offs

The most effective mitigation is targeted data collection from the missing groups, but practitioners combine several techniques when new data is expensive or slow to obtain. Each approach has costs that must be weighed.

ApproachHow it helpsMain risk
Targeted collectionAdds real examples from missing groupsCost, consent and privacy obligations
Re-samplingOversamples minority records in trainingOverfitting to duplicated examples
Re-weightingGives minority records larger loss weightHigher variance in learned patterns
Synthetic dataGenerates extra examples for rare groupsArtifacts that do not match reality
Group-aware evaluationSets minimum performance per subgroupRequires reliable group labels
Mitigation options for representation bias

Re-sampling and re-weighting can balance the training signal but cannot invent information that the data never contained. When a group shows genuinely different patterns, such as different symptom presentation in a medical condition, only new, real data reliably closes the gap.

Consequences and common misconceptions

The practical consequence of representation bias is unequal error rates, which can turn into allocative harm when people are denied services and into representational harm when outputs ignore or caricature them. Several misconceptions make the problem harder to address.

MisconceptionReality
A large dataset is automatically representativeSize does not fix a skewed sampling frame
High overall accuracy means a fair modelAverages can hide poor subgroup results
Removing protected attributes removes biasUnder-represented groups still get worse predictions
The issue only affects demographic groupsRare conditions, regions and devices are affected too
Misconceptions about representation bias

Regulation increasingly addresses the issue directly. The EU AI Act requires that training, validation and test data for high-risk AI systems be relevant and sufficiently representative, which turns data composition from a good practice into a documented compliance duty.

In simple terms

An AI system learns from examples, much like a student learns from a textbook. If the textbook covers some people in detail and barely mentions others, the student will know much about some and very little about the rest.

Representation bias is that uneven textbook. The AI may look excellent on average while making far more mistakes for people who were rare in its training data. The fix is not a cleverer student but a better, more complete textbook.

A practical example

A hospital network develops a skin lesion classifier to help general practitioners decide which patients to refer to a dermatologist. The training set contains about 40,000 images, but fewer than 5 percent show darker skin tones, because most images came from clinics serving mainly lighter-skinned patients.

Overall sensitivity for melanoma is 91 percent, which looks strong. When the validation team splits results by skin tone, sensitivity for the darkest skin types drops to about 70 percent, meaning many cancers in those patients would be missed.

The team pauses deployment, partners with clinics serving more diverse populations to collect several thousand additional consented images, and sets a minimum sensitivity for every skin tone group as a release condition. After retraining, the gap narrows to a few percentage points, and the remaining limitation is stated openly in the model documentation.