SHAP (SHapley Additive exPlanations) is a method for explaining individual predictions of a machine learning model by assigning each input feature a Shapley value: its fair share of the difference between that prediction and the model’s average prediction. In AI ethics, SHAP is one of the most widely used tools for transparency, auditing and contestability, because it lets developers, auditors and affected people see which factors pushed a decision up or down.
SHAP was introduced in 2017 as a unifying framework that connects cooperative game theory with earlier attribution methods such as LIME. Its appeal comes from formal guarantees that most explanation methods lack, while its limitations come from the fact that it explains the model, not the world.
How SHAP distributes a prediction among features
SHAP treats the features of one input as players in a cooperative game whose payout is the model prediction. The Shapley value of a feature is its average marginal contribution across all possible orders in which features could be added to a coalition, which is the unique allocation rule that satisfies a small set of fairness axioms.
The result is an additive explanation: the base value, which is the expected model output over a background dataset, plus the sum of all feature SHAP values equals the actual prediction. For a credit model with an average approval score of 0.40 and an applicant score of 0.72, the SHAP values of all features sum to exactly 0.32.
The axioms below explain why SHAP is often preferred where explanations must be defensible.
| Property | Meaning | Why it matters |
|---|---|---|
| Local accuracy | Attributions sum to the prediction minus the base value | No part of the output is left unexplained |
| Missingness | A feature absent from the input receives zero | No credit for information the model never saw |
| Consistency | If a feature matters more in a new model, its value does not fall | Comparisons between models stay meaningful |
| Symmetry | Features with identical contributions receive equal values | No arbitrary favoritism among features |
SHAP estimators and when each applies
Exact Shapley values require evaluating the model on every subset of features, which grows as 2 to the power of the feature count and is infeasible beyond about 20 features. SHAP therefore relies on model-specific estimators that trade generality for speed.
| Estimator | Model types | Speed | Key caveat |
|---|---|---|---|
| TreeSHAP | Decision trees, random forests, gradient boosting | Fast, exact for trees | Path-dependent variant reflects tree structure, not only data |
| KernelSHAP | Any model | Slow, sampling-based | Assumes feature independence when masking |
| LinearSHAP | Linear and logistic models | Very fast, closed form | Only for linear structure |
| DeepSHAP | Neural networks | Moderate | Approximation built on DeepLIFT rules |
| Permutation explainer | Any model | Moderate | Variance depends on the number of samples |
The background dataset is a design decision with ethical weight. Explaining a loan denial relative to all applicants yields different attributions than explaining it relative to approved applicants, so the reference population should be documented and justified in any audit.
SHAP compared with other explanation methods
SHAP is a post-hoc, model-agnostic attribution method that produces both local explanations for single predictions and global summaries by aggregating many local values. The comparison below positions it among neighboring techniques in the same field.
| Method | Scope | Output | Main trade-off |
|---|---|---|---|
| SHAP | Local and global | Additive feature attributions | Strong guarantees, high computational cost |
| LIME | Local | Weights of a local surrogate model | Fast but unstable across runs |
| Counterfactual explanation | Local | Minimal change that flips the outcome | Actionable but not a full attribution |
| Partial dependence plot | Global | Average effect of one feature | Hides heterogeneity between individuals |
| Integrated gradients | Local | Attributions for differentiable models | Requires gradients and a baseline input |
For affected individuals, SHAP answers “why this score?”, whereas a counterfactual answers “what would change it?”. Fairness reviews often combine both, because a large attribution does not tell a person what action is realistic.
Using SHAP in fairness audits
In AI ethics practice, SHAP is used to detect reliance on protected attributes and their proxies, to document model behavior for regulators and to support explanations owed to affected people. Global summary plots show which features dominate, and dependence plots reveal how a feature’s effect varies across groups.
- Compare SHAP distributions across groups: a postal code feature with large attributions for one ethnic group may signal proxy discrimination.
- Inspect interaction values to see whether a feature such as age changes the effect of income.
- Check stability by recomputing values with different background samples and random seeds.
- Translate attributions into plain reasons before showing them to customers or applicants.
SHAP supports transparency obligations, such as reasons for adverse credit decisions or documentation duties for high-risk systems, but no regulation mandates SHAP specifically. It is one acceptable technique among several, and its outputs still require human interpretation and context.
Limitations and common misreadings
The most frequent error is reading SHAP values as causal effects in the real world. A high attribution means the model relied on a feature, not that changing the feature would change the outcome for a person, especially when features are strongly correlated.
| Pitfall | Cause | Safer practice |
|---|---|---|
| Causal reading | Attributions describe the model, not reality | Use causal analysis for interventions |
| Split credit among correlated features | Masking assumes independence | Group correlated features or use conditional SHAP |
| Unrealistic inputs | KernelSHAP samples impossible feature combinations | Choose a representative background set |
| False reassurance on fairness | Low attribution to a protected attribute hides proxies | Combine with group fairness metrics |
| Manipulable explanations | Models can be built to hide bias from sampling explainers | Audit with several methods and real data |
Research has shown that adversarially designed models can mask discriminatory behavior from perturbation-based explainers, which is a direct concern for ethics washing. SHAP is therefore strongest as one layer of an audit, alongside fairness metrics, data documentation and outcome monitoring.
The idea in brief
SHAP is a way to ask a computer model why it gave a particular answer. Imagine a team that wins a prize. SHAP works out how much each player really contributed. Then it shares the prize fairly.
For a model, the players are the facts about a person, such as income or debt. SHAP shows which facts raised the score and which lowered it. It shows how the model reasoned. It does not prove that the reasoning was fair or correct.
A practical scenario
A regional bank uses a gradient boosting model to score small business loan applications. Before renewing the model, its internal model risk team runs a fairness review with TreeSHAP on 40,000 past applications, using the full applicant pool as the background dataset.
The global summary shows that debt service coverage and payment history dominate, as expected. However, the dependence plots reveal that the distance of the business from the city center carries large negative attributions, concentrated in districts with a high share of minority-owned firms. The reviewers conclude that the feature acts as a likely proxy for a protected attribute and confirm the concern with a disparate impact ratio of 0.74, below the commonly used 0.80 threshold.
The team removes the distance feature, retrains the model and repeats the analysis. Predictive accuracy falls by less than one percentage point, the disparity ratio rises above 0.80, and the bank uses the top three SHAP attributions of each declined application as the basis for its written adverse action reasons.