Equalized odds is a group fairness criterion for classification models that requires the true positive rate and the false positive rate to be equal across groups defined by a protected attribute, such as sex or ethnicity. Formally, the prediction must be conditionally independent of the protected attribute given the true outcome, so that people who share the same actual label receive positive predictions at the same rate regardless of group.
In AI ethics, equalized odds is one of the most cited fairness metrics because it accepts that base rates may differ between groups while still demanding that errors are distributed evenly. It was formalized in 2016 by Hardt, Price and Srebro, who also proposed a post-processing method to achieve it.
How equalized odds is defined
Equalized odds compares two conditional error rates across groups. For a binary classifier with prediction Ŷ, true label Y and protected attribute A, the criterion holds when P(Ŷ = 1 | Y = y, A = a) is the same for every group a, both for y = 1 and for y = 0.
The condition for y = 1 equalizes the true positive rate, also called sensitivity or recall. The condition for y = 0 equalizes the false positive rate, which also fixes the true negative rate. The table below shows the quantities involved.
| Quantity | Definition | Constrained across groups |
|---|---|---|
| True positive rate | Share of actual positives predicted positive | Yes |
| False positive rate | Share of actual negatives predicted positive | Yes |
| False negative rate | Share of actual positives predicted negative | Yes, as 1 minus TPR |
| Positive prediction rate | Share of all people predicted positive | No |
| Precision | Share of positive predictions that are correct | No |
Because the criterion conditions on the true label, it requires reliable ground truth for every group. If the labels themselves reflect historical bias or measurement bias, equalized odds can certify a model as fair while it reproduces a biased target variable.
Equalized odds compared with other fairness metrics
Equalized odds sits between weaker and stricter notions of group fairness. Equal opportunity is its relaxed version, demographic parity ignores the true label entirely, and predictive parity conditions on the prediction instead of the outcome.
| Metric | What must be equal | Uses true labels |
|---|---|---|
| Equalized odds | TPR and FPR | Yes |
| Equal opportunity | TPR only | Yes |
| Demographic parity | Positive prediction rate | No |
| Predictive parity | Precision (PPV) | Yes |
| Calibration within groups | Observed rate at each score | Yes |
An important result, known as the impossibility theorem of fairness, shows that when base rates differ between groups, a classifier that is not perfect cannot satisfy equalized odds and predictive parity at the same time. This explains the well-known disagreement around recidivism risk scores, where one side measured error rate balance and the other measured calibration.
Methods for achieving equalized odds
Equalized odds can be enforced at three stages of the machine learning pipeline. Each stage offers different control over accuracy, legal exposure and operational complexity.
| Stage | Typical technique | Main drawback |
|---|---|---|
| Pre-processing | Reweighting or resampling training data | Only indirect control over error rates |
| In-processing | Constrained optimization or adversarial training | More complex training and tuning |
| Post-processing | Group-specific thresholds with randomization | Requires the protected attribute at decision time |
The original post-processing approach chooses, for each group, a point on that group’s ROC curve where the curves overlap, sometimes randomizing between two thresholds to hit the exact rates. It needs no retraining, but explicit use of the protected attribute at decision time may be restricted in some jurisdictions.
In practice, exact equality is rare, so teams set a tolerance on the maximum gap, for example a difference of a few percentage points in TPR and FPR, and report the equalized odds difference as the larger of the two gaps.
When equalized odds is the right choice
Equalized odds fits situations where both types of error cause real harm and where outcome labels are trustworthy. It is less suitable when only one error type matters or when the labels themselves are contested.
| Situation | Recommended criterion | Reason |
|---|---|---|
| Both false alarms and misses harm people | Equalized odds | Balances both error types |
| Only missed qualified candidates matter | Equal opportunity | Focuses on true positives |
| Labels reflect past discrimination | Demographic parity or label review | Ground truth is unreliable |
| Scores are read as probabilities | Calibration within groups | Preserves score meaning |
Typical domains include medical screening, fraud detection and content moderation, where a false positive burdens an innocent person and a false negative leaves a real case unaddressed.
Limitations and common misconceptions
Equalized odds is a statistical property of groups, not a guarantee of fair treatment for individuals. Two people with identical relevant features may still receive different outcomes, which is the concern of individual fairness.
- Satisfying it on test data does not ensure it holds after deployment, because data drift changes error rates.
- Intersectional groups may violate the criterion even when every single attribute satisfies it.
- Enforcing it usually costs accuracy, reflecting the fairness-accuracy trade-off, and can lower performance for all groups.
- Small groups produce noisy estimates, so confidence intervals should accompany every reported gap.
Equalized odds is therefore best used as one input to an AI audit, alongside label quality analysis, stakeholder consultation and an algorithmic impact assessment.
Stripped to the essentials
Equalized odds is a fairness test for systems that sort people into yes or no. It asks a simple question. Does the system make the same kinds of mistakes for every group?
Think of a smoke alarm installed in two apartments. A fair alarm should catch real fires equally often in both. It should also raise false alarms equally often in both. Equalized odds demands exactly that balance. The key idea is that errors must be shared evenly, not pushed onto one group.
Where it is applied
A hospital network deploys a model that flags patients at high risk of sepsis for early review. An internal audit finds that the model detects 85 percent of true sepsis cases among men but only 74 percent among women, while its false alarm rate is similar for both.
The data science team measures an equalized odds difference of 11 percentage points, driven entirely by the true positive gap. After reviewing whether diagnosis codes were recorded consistently, the team retrains the model with a constraint on error rate gaps and adjusts the alert threshold.
The revised model reaches about 82 percent sensitivity for both groups, with false alarm rates within two points of each other. Overall accuracy falls slightly, but the hospital accepts this trade-off because missed sepsis cases among women were the more serious clinical risk. Monitoring continues monthly to detect drift.