Disparate impact

Also known as: adverse impact

On this page 7

Disparate impact is a form of discrimination in which a facially neutral rule, test or model produces outcomes that fall disproportionately on a protected group, even though nobody intended to treat that group differently. In AI ethics, the concept is central because machine learning systems rarely use protected attributes openly, yet they can still reproduce group disparities through correlated features and biased historical data. The idea is also known as adverse impact, the term preferred in employment testing.

The concept originated in United States anti-discrimination law and has become a standard lens for evaluating automated decisions in hiring, lending, housing, insurance and public services. It shifts attention from intent to measurable outcomes: what matters is who is harmed, by how much, and whether the practice can be justified.

Disparate impact doctrine was established by the United States Supreme Court in Griggs v. Duke Power Co. (1971), which held that employment requirements with a discriminatory effect are unlawful unless they are related to the job and consistent with business necessity. The Civil Rights Act of 1991 later codified the doctrine in Title VII.

Courts apply a burden-shifting framework, summarized in the table below, which also shapes how organizations document and defend algorithmic decisions.

StepWho carries the burdenWhat must be shown
Prima facie casePlaintiffA specific practice causes a significant disparity
Business necessityEmployer or providerThe practice is job-related and necessary
Less discriminatory alternativePlaintiffAn equally effective option with less disparity exists
The disparate impact burden-shifting framework

Similar principles appear in the Fair Housing Act and the Equal Credit Opportunity Act in the United States, and European law covers comparable ground under the name indirect discrimination, defined in the EU equal treatment directives.

How disparate impact arises in AI systems

AI systems create disparate impact mainly through data and feature correlations, not through explicit rules. Removing the protected attribute, an approach called fairness through unawareness, seldom solves the problem because other variables carry the same information.

The table below lists the most common mechanisms and how each one transmits group disparities into model outputs.

MechanismHow it produces disparityExample
Proxy featuresNeutral variables correlate with group membershipPostal code standing in for ethnicity
Historical biasLabels encode past discriminatory decisionsPast hiring choices used as targets
Representation biasA group is underrepresented in training dataHigher error rates for minority speakers
Measurement biasThe target is measured differently across groupsArrests used as a proxy for crime
Threshold choiceOne cutoff applied to differently shaped score distributionsCredit score cutoffs in lending
Common sources of disparate impact in machine learning

These mechanisms often interact. A model can be accurate overall and still produce unequal selection rates, because accuracy measures agreement with labels that may themselves reflect earlier disparities.

Measuring disparate impact

Disparate impact is usually measured by comparing selection rates or favorable outcome rates between groups. The most widely cited heuristic is the four-fifths rule from the Uniform Guidelines on Employee Selection Procedures (1978), described in detail in its own entry.

The disparate impact ratio divides the favorable outcome rate of the less favored group by that of the most favored group. A ratio below 0.8 is treated as preliminary evidence of adverse impact, although courts and regulators also look at statistical significance and practical size.

MetricCalculationInterpretation
Impact ratioRate of group A divided by rate of group BBelow 0.8 signals concern
Statistical parity differenceRate of group A minus rate of group BZero indicates equal rates
Two-standard-deviation testSignificance of the observed gapTests whether chance explains it
Shortfall analysisExpected minus actual selectionsShows the practical size of the gap
Metrics used to quantify disparate impact

Each metric has blind spots. Ratios become unstable with small samples, and significance tests flag trivial gaps in very large datasets, so practitioners report several measures together.

Disparate impact is a legal and outcome-based concept, whereas most fairness metrics in machine learning are mathematical criteria. Several of them, especially demographic parity, are closely related but not identical.

ConceptFocusRelationship to disparate impact
Disparate treatmentIntentional differential treatmentThe intent-based counterpart
Demographic parityEqual positive rates across groupsThe metric closest to impact ratios
Equalized oddsEqual error rates across groupsCan hold while selection rates differ
Allocative harmUnfair distribution of resourcesThe type of harm disparate impact describes
Disparate impact and neighboring concepts

A tension exists between the two legal doctrines. Correcting disparate impact by adjusting outcomes based on group membership can itself raise disparate treatment concerns, which is why remedies usually target features, data or thresholds rather than explicit group quotas.

Mitigation and common mistakes

Effective mitigation starts with auditing outcomes before deployment and continues with monitoring after launch, because population shifts can create new disparities. Techniques fall into pre-processing, in-processing and post-processing stages, discussed under bias mitigation.

  • Dropping protected attributes is not enough: proxies keep the disparity alive.
  • Ignoring the business necessity test: a model needs documented validity for its purpose.
  • Not searching for alternatives: a less discriminatory model with similar accuracy often exists.
  • Auditing only once: data drift can push ratios below thresholds months after launch.
  • Lacking demographic data: without it, organizations cannot measure impact at all.

Many regulations now expect this discipline. New York City Local Law 144, for example, requires annual bias audits of automated employment decision tools with published impact ratios.

The idea in brief

Disparate impact means a rule that looks fair on paper ends up hurting one group much more than others. Nobody has to intend harm for it to happen.

Imagine a door that is the same height for everyone but is too low for tall people. The door treats everyone identically, yet some people keep bumping their heads. With AI, the door is the model, and checking who gets hurt is the only way to notice the problem.

A practical scenario

A regional bank tests a machine learning model for small personal loans before rollout. The model never sees applicants’ ethnicity, but the validation team, using surveyed demographic data, compares approval rates across groups.

Approval reaches 62 percent for the most favored group and 44 percent for another group, an impact ratio of about 0.71, below the four-fifths benchmark. Feature analysis shows that length of residence at the current address and a neighborhood-level variable carry much of the gap.

The team replaces the neighborhood variable with direct measures of repayment capacity and retrains. Predictive accuracy drops by less than one percentage point, while the impact ratio rises to 0.86. The bank documents the analysis, the alternative considered and the final choice, and schedules quarterly monitoring of approval rates.