SHAP

Also known as: SHapley Additive exPlanations

On this page 7

SHAP (SHapley Additive exPlanations) is a method for explaining individual predictions of a machine learning model by assigning each input feature a Shapley value: its fair share of the difference between that prediction and the model’s average prediction. In AI ethics, SHAP is one of the most widely used tools for transparency, auditing and contestability, because it lets developers, auditors and affected people see which factors pushed a decision up or down.

SHAP was introduced in 2017 as a unifying framework that connects cooperative game theory with earlier attribution methods such as LIME. Its appeal comes from formal guarantees that most explanation methods lack, while its limitations come from the fact that it explains the model, not the world.

How SHAP distributes a prediction among features

SHAP treats the features of one input as players in a cooperative game whose payout is the model prediction. The Shapley value of a feature is its average marginal contribution across all possible orders in which features could be added to a coalition, which is the unique allocation rule that satisfies a small set of fairness axioms.

The result is an additive explanation: the base value, which is the expected model output over a background dataset, plus the sum of all feature SHAP values equals the actual prediction. For a credit model with an average approval score of 0.40 and an applicant score of 0.72, the SHAP values of all features sum to exactly 0.32.

The axioms below explain why SHAP is often preferred where explanations must be defensible.

PropertyMeaningWhy it matters
Local accuracyAttributions sum to the prediction minus the base valueNo part of the output is left unexplained
MissingnessA feature absent from the input receives zeroNo credit for information the model never saw
ConsistencyIf a feature matters more in a new model, its value does not fallComparisons between models stay meaningful
SymmetryFeatures with identical contributions receive equal valuesNo arbitrary favoritism among features
Properties that SHAP values satisfy

SHAP estimators and when each applies

Exact Shapley values require evaluating the model on every subset of features, which grows as 2 to the power of the feature count and is infeasible beyond about 20 features. SHAP therefore relies on model-specific estimators that trade generality for speed.

EstimatorModel typesSpeedKey caveat
TreeSHAPDecision trees, random forests, gradient boostingFast, exact for treesPath-dependent variant reflects tree structure, not only data
KernelSHAPAny modelSlow, sampling-basedAssumes feature independence when masking
LinearSHAPLinear and logistic modelsVery fast, closed formOnly for linear structure
DeepSHAPNeural networksModerateApproximation built on DeepLIFT rules
Permutation explainerAny modelModerateVariance depends on the number of samples
Common SHAP estimators

The background dataset is a design decision with ethical weight. Explaining a loan denial relative to all applicants yields different attributions than explaining it relative to approved applicants, so the reference population should be documented and justified in any audit.

SHAP compared with other explanation methods

SHAP is a post-hoc, model-agnostic attribution method that produces both local explanations for single predictions and global summaries by aggregating many local values. The comparison below positions it among neighboring techniques in the same field.

MethodScopeOutputMain trade-off
SHAPLocal and globalAdditive feature attributionsStrong guarantees, high computational cost
LIMELocalWeights of a local surrogate modelFast but unstable across runs
Counterfactual explanationLocalMinimal change that flips the outcomeActionable but not a full attribution
Partial dependence plotGlobalAverage effect of one featureHides heterogeneity between individuals
Integrated gradientsLocalAttributions for differentiable modelsRequires gradients and a baseline input
SHAP and related explainability techniques

For affected individuals, SHAP answers “why this score?”, whereas a counterfactual answers “what would change it?”. Fairness reviews often combine both, because a large attribution does not tell a person what action is realistic.

Using SHAP in fairness audits

In AI ethics practice, SHAP is used to detect reliance on protected attributes and their proxies, to document model behavior for regulators and to support explanations owed to affected people. Global summary plots show which features dominate, and dependence plots reveal how a feature’s effect varies across groups.

  • Compare SHAP distributions across groups: a postal code feature with large attributions for one ethnic group may signal proxy discrimination.
  • Inspect interaction values to see whether a feature such as age changes the effect of income.
  • Check stability by recomputing values with different background samples and random seeds.
  • Translate attributions into plain reasons before showing them to customers or applicants.

SHAP supports transparency obligations, such as reasons for adverse credit decisions or documentation duties for high-risk systems, but no regulation mandates SHAP specifically. It is one acceptable technique among several, and its outputs still require human interpretation and context.

Limitations and common misreadings

The most frequent error is reading SHAP values as causal effects in the real world. A high attribution means the model relied on a feature, not that changing the feature would change the outcome for a person, especially when features are strongly correlated.

PitfallCauseSafer practice
Causal readingAttributions describe the model, not realityUse causal analysis for interventions
Split credit among correlated featuresMasking assumes independenceGroup correlated features or use conditional SHAP
Unrealistic inputsKernelSHAP samples impossible feature combinationsChoose a representative background set
False reassurance on fairnessLow attribution to a protected attribute hides proxiesCombine with group fairness metrics
Manipulable explanationsModels can be built to hide bias from sampling explainersAudit with several methods and real data
Pitfalls when interpreting SHAP

Research has shown that adversarially designed models can mask discriminatory behavior from perturbation-based explainers, which is a direct concern for ethics washing. SHAP is therefore strongest as one layer of an audit, alongside fairness metrics, data documentation and outcome monitoring.

The idea in brief

SHAP is a way to ask a computer model why it gave a particular answer. Imagine a team that wins a prize. SHAP works out how much each player really contributed. Then it shares the prize fairly.

For a model, the players are the facts about a person, such as income or debt. SHAP shows which facts raised the score and which lowered it. It shows how the model reasoned. It does not prove that the reasoning was fair or correct.

A practical scenario

A regional bank uses a gradient boosting model to score small business loan applications. Before renewing the model, its internal model risk team runs a fairness review with TreeSHAP on 40,000 past applications, using the full applicant pool as the background dataset.

The global summary shows that debt service coverage and payment history dominate, as expected. However, the dependence plots reveal that the distance of the business from the city center carries large negative attributions, concentrated in districts with a high share of minority-owned firms. The reviewers conclude that the feature acts as a likely proxy for a protected attribute and confirm the concern with a disparate impact ratio of 0.74, below the commonly used 0.80 threshold.

The team removes the distance feature, retrains the model and repeats the analysis. Predictive accuracy falls by less than one percentage point, the disparity ratio rises above 0.80, and the bank uses the top three SHAP attributions of each declined application as the basis for its written adverse action reasons.