Algorithmic fairness

Also known as: AI fairness · fairness in machine learning

On this page 8

Algorithmic fairness is the field that defines, measures and corrects systematic disadvantage in automated decisions, so that a model does not treat people worse because of protected traits such as race, sex, age or disability. Within AI ethics and artificial intelligence more broadly, it turns the social idea of equal treatment into formal criteria that can be tested on data.

The field is also called AI fairness or fairness in machine learning. Its central insight is that fairness has several mathematical definitions that often conflict, so choosing one is a value judgment about the decision context, not a purely technical step.

Where unfairness in algorithms comes from

Unfairness in algorithms usually comes from data and problem design rather than from malicious code. A model learns patterns from past outcomes, and if those outcomes reflect unequal treatment, the model reproduces and can even amplify them.

Removing the protected attribute rarely solves the problem. Variables such as postal code, school or purchase history can act as proxies for a protected trait, so a model can discriminate without ever seeing it. This is why fairness through unawareness is widely considered insufficient.

The table below lists the main sources of bias and how each one enters a system.

SourceHow it enters the modelTypical example
Historical biasPast decisions encode inequalityHiring data favoring one group
Representation biasSome groups are underrepresentedFace data with few darker-skinned faces
Measurement biasThe label is a flawed proxy for the goalArrests used as a proxy for crime
Proxy discriminationCorrelated features stand in for a traitPostal code standing in for ethnicity
Feedback loopsModel outputs shape future training dataPredictive policing in the same areas
Common sources of algorithmic bias

Main definitions of algorithmic fairness

Algorithmic fairness has two broad families of definitions: group fairness, which compares statistics between groups, and individual fairness, which requires that similar individuals receive similar outcomes. A third approach, counterfactual fairness, asks whether a decision would change if only the protected attribute were different, using a causal model.

The most widely used group criteria are compared below. Each is a separate fairness metric with its own Term page.

CriterionWhat must be equal across groupsTypical concern
Demographic parityRate of positive decisionsIgnores real differences in base rates
Equal opportunityTrue positive rateAllows unequal false positives
Equalized oddsTrue and false positive ratesOften lowers overall accuracy
Predictive parityPrecision of positive predictionsCan permit unequal error rates
CalibrationMeaning of a given risk scoreCompatible with unequal error rates
Widely used group fairness criteria

In employment screening, the four-fifths rule from United States practice serves as a rough test of disparate impact: a selection rate for one group below 80 percent of the highest group’s rate signals possible adverse impact that needs investigation.

Why fairness criteria conflict

Fairness criteria conflict because groups often have different base rates for the predicted outcome. Well-known impossibility results show that when base rates differ and the predictor is imperfect, a model cannot satisfy calibration and equal error rates at the same time.

This was the core of a widely debated case about recidivism scores: one analysis showed unequal false positive rates between groups, while the developer showed the scores were equally calibrated. Both claims could be true simultaneously, which illustrates that fairness disputes are often disagreements about which criterion matters.

There is also a fairness-accuracy trade-off. Enforcing a constraint usually costs some overall accuracy, although the loss is often small, and the accuracy being measured may itself rest on biased labels.

Techniques for improving fairness

Fairness can be improved at three points in the machine learning pipeline: before training, during training and after training. The right choice depends on what the team can change and on legal limits on using protected attributes.

The table below compares these bias mitigation stages.

StageExample methodsMain trade-off
Pre-processingReweighting, resampling, relabeling dataMay distort real patterns
In-processingFairness constraints, adversarial debiasingRequires access to training
Post-processingGroup-specific decision thresholdsMay be legally contested
Bias mitigation by pipeline stage

Technical methods work only alongside governance practices: documenting data with datasheets, running an algorithmic impact assessment before deployment, conducting independent AI audits and giving affected people a route to contest decisions.

Choosing a fairness approach in practice

Choosing a fairness approach means matching the criterion to the harm that matters most in the specific decision. A lending model, a medical triage tool and a content filter call for different priorities.

Decision contextPriority harmSuitable criterion
Loan approvalQualified applicants wrongly rejectedEqual opportunity
Pretrial risk scoringUnequal wrongful detentionEqualized odds
Medical risk scoresMisleading probabilitiesCalibration within groups
Job ad deliveryUnequal access to opportunitiesDemographic parity
Matching criteria to decision contexts

Regulation increasingly shapes this choice. The EU AI Act classifies credit scoring, recruitment and similar uses as high-risk systems, requiring data governance and bias monitoring, while anti-discrimination law in many countries already applies to automated decisions.

Common misconceptions

  • A single fairness score exists: in reality several metrics exist and they can disagree.
  • Deleting sensitive attributes makes a model fair: proxies often carry the same information.
  • An accurate model is automatically fair: high overall accuracy can hide large errors for small groups.
  • Fairness is solved once: data drift and feedback loops require continuous monitoring after deployment.

Put simply

Algorithmic fairness is about making sure computer decisions treat people justly. Computers learn from past records. If those records were unfair, the computer can repeat the same unfairness.

People disagree on what fair means. Should every group get the same share of approvals? Or should every group face the same chance of a wrong decision? Both ideas sound right, but they often cannot both hold. The key takeaway is that fairness needs a clear choice, careful testing and human oversight.

In practice

A regional bank builds a machine learning model to approve personal loans. Before launch, the risk team tests it on 50,000 past applications and finds that qualified applicants from one ethnic group are rejected noticeably more often than equally qualified applicants from others. The model never used ethnicity, but postal code and employment history acted as proxies.

The team chooses equal opportunity as its main criterion, because the key harm is denying credit to people who would repay. It reweights the training data, removes the strongest proxy feature and rechecks results by group. The gap in approval of qualified applicants falls to within two percentage points, while overall default prediction loses less than one point of accuracy.

The bank documents the decision in an impact assessment, adds quarterly fairness monitoring and offers rejected applicants a human review on request.