Disparate impact is a form of discrimination in which a facially neutral rule, test or model produces outcomes that fall disproportionately on a protected group, even though nobody intended to treat that group differently. In AI ethics, the concept is central because machine learning systems rarely use protected attributes openly, yet they can still reproduce group disparities through correlated features and biased historical data. The idea is also known as adverse impact, the term preferred in employment testing.
The concept originated in United States anti-discrimination law and has become a standard lens for evaluating automated decisions in hiring, lending, housing, insurance and public services. It shifts attention from intent to measurable outcomes: what matters is who is harmed, by how much, and whether the practice can be justified.
Legal origins of disparate impact
Disparate impact doctrine was established by the United States Supreme Court in Griggs v. Duke Power Co. (1971), which held that employment requirements with a discriminatory effect are unlawful unless they are related to the job and consistent with business necessity. The Civil Rights Act of 1991 later codified the doctrine in Title VII.
Courts apply a burden-shifting framework, summarized in the table below, which also shapes how organizations document and defend algorithmic decisions.
| Step | Who carries the burden | What must be shown |
|---|---|---|
| Prima facie case | Plaintiff | A specific practice causes a significant disparity |
| Business necessity | Employer or provider | The practice is job-related and necessary |
| Less discriminatory alternative | Plaintiff | An equally effective option with less disparity exists |
Similar principles appear in the Fair Housing Act and the Equal Credit Opportunity Act in the United States, and European law covers comparable ground under the name indirect discrimination, defined in the EU equal treatment directives.
How disparate impact arises in AI systems
AI systems create disparate impact mainly through data and feature correlations, not through explicit rules. Removing the protected attribute, an approach called fairness through unawareness, seldom solves the problem because other variables carry the same information.
The table below lists the most common mechanisms and how each one transmits group disparities into model outputs.
| Mechanism | How it produces disparity | Example |
|---|---|---|
| Proxy features | Neutral variables correlate with group membership | Postal code standing in for ethnicity |
| Historical bias | Labels encode past discriminatory decisions | Past hiring choices used as targets |
| Representation bias | A group is underrepresented in training data | Higher error rates for minority speakers |
| Measurement bias | The target is measured differently across groups | Arrests used as a proxy for crime |
| Threshold choice | One cutoff applied to differently shaped score distributions | Credit score cutoffs in lending |
These mechanisms often interact. A model can be accurate overall and still produce unequal selection rates, because accuracy measures agreement with labels that may themselves reflect earlier disparities.
Measuring disparate impact
Disparate impact is usually measured by comparing selection rates or favorable outcome rates between groups. The most widely cited heuristic is the four-fifths rule from the Uniform Guidelines on Employee Selection Procedures (1978), described in detail in its own entry.
The disparate impact ratio divides the favorable outcome rate of the less favored group by that of the most favored group. A ratio below 0.8 is treated as preliminary evidence of adverse impact, although courts and regulators also look at statistical significance and practical size.
| Metric | Calculation | Interpretation |
|---|---|---|
| Impact ratio | Rate of group A divided by rate of group B | Below 0.8 signals concern |
| Statistical parity difference | Rate of group A minus rate of group B | Zero indicates equal rates |
| Two-standard-deviation test | Significance of the observed gap | Tests whether chance explains it |
| Shortfall analysis | Expected minus actual selections | Shows the practical size of the gap |
Each metric has blind spots. Ratios become unstable with small samples, and significance tests flag trivial gaps in very large datasets, so practitioners report several measures together.
Disparate impact compared with related fairness concepts
Disparate impact is a legal and outcome-based concept, whereas most fairness metrics in machine learning are mathematical criteria. Several of them, especially demographic parity, are closely related but not identical.
| Concept | Focus | Relationship to disparate impact |
|---|---|---|
| Disparate treatment | Intentional differential treatment | The intent-based counterpart |
| Demographic parity | Equal positive rates across groups | The metric closest to impact ratios |
| Equalized odds | Equal error rates across groups | Can hold while selection rates differ |
| Allocative harm | Unfair distribution of resources | The type of harm disparate impact describes |
A tension exists between the two legal doctrines. Correcting disparate impact by adjusting outcomes based on group membership can itself raise disparate treatment concerns, which is why remedies usually target features, data or thresholds rather than explicit group quotas.
Mitigation and common mistakes
Effective mitigation starts with auditing outcomes before deployment and continues with monitoring after launch, because population shifts can create new disparities. Techniques fall into pre-processing, in-processing and post-processing stages, discussed under bias mitigation.
- Dropping protected attributes is not enough: proxies keep the disparity alive.
- Ignoring the business necessity test: a model needs documented validity for its purpose.
- Not searching for alternatives: a less discriminatory model with similar accuracy often exists.
- Auditing only once: data drift can push ratios below thresholds months after launch.
- Lacking demographic data: without it, organizations cannot measure impact at all.
Many regulations now expect this discipline. New York City Local Law 144, for example, requires annual bias audits of automated employment decision tools with published impact ratios.
The idea in brief
Disparate impact means a rule that looks fair on paper ends up hurting one group much more than others. Nobody has to intend harm for it to happen.
Imagine a door that is the same height for everyone but is too low for tall people. The door treats everyone identically, yet some people keep bumping their heads. With AI, the door is the model, and checking who gets hurt is the only way to notice the problem.
A practical scenario
A regional bank tests a machine learning model for small personal loans before rollout. The model never sees applicants’ ethnicity, but the validation team, using surveyed demographic data, compares approval rates across groups.
Approval reaches 62 percent for the most favored group and 44 percent for another group, an impact ratio of about 0.71, below the four-fifths benchmark. Feature analysis shows that length of residence at the current address and a neighborhood-level variable carry much of the gap.
The team replaces the neighborhood variable with direct measures of repayment capacity and retrains. Predictive accuracy drops by less than one percentage point, while the impact ratio rises to 0.86. The bank documents the analysis, the alternative considered and the final choice, and schedules quarterly monitoring of approval rates.