Post-hoc explanation

Also known as: post-hoc explainability

On this page 7

A post-hoc explanation is an account of a trained model’s behavior produced after training, by a separate method that analyzes the model’s inputs, outputs or internals without changing how the model makes decisions. In AI ethics, post-hoc explainability is a central tool for transparency and accountability, because many high-performing systems are black-box models whose reasoning cannot be read directly.

Post-hoc methods answer questions such as “which features drove this loan denial” or “what would have to change for a different outcome”. Their value is real but limited: an explanation is an approximation of the model, not the model itself, and it can be incomplete, unstable or misleading if it is used without care.

How post-hoc explanation methods work

Post-hoc explanation methods probe a fixed model and summarize what they observe in a form people can inspect. Some methods are model-agnostic and treat the model as a function to query, perturbing inputs and watching outputs. Others are model-specific and use internal quantities such as gradients or attention weights, which requires access to the model’s parameters.

A second distinction is scope. Local explanations describe a single prediction, for example one applicant’s credit score, while global explanations describe the overall behavior of the model across a dataset. A third distinction is the form of the output: feature importance scores, highlighted image regions, example cases, simplified rules or counterfactual changes.

The table below classifies widely used methods along these dimensions.

MethodScopeModel accessOutput
SHAPLocal and globalAgnostic or specificAdditive feature attributions
LIMELocalAgnosticWeights of a local surrogate model
Integrated gradientsLocalGradients requiredAttributions along an input path
Grad-CAMLocalConvolutional layers requiredHeat map over image regions
Partial dependence plotGlobalAgnosticAverage effect of one feature
Counterfactual explanationLocalAgnosticMinimal input change that flips the outcome
Major families of post-hoc methods

Post-hoc explanation versus interpretable models

The main alternative to post-hoc explanation is an interpretable model, such as a short decision tree, a sparse linear model or a scoring system, whose structure is itself the explanation. The choice between the two approaches is one of the most debated questions in responsible AI, especially for high-stakes decisions in lending, hiring, health care and criminal justice.

AspectPost-hoc explanationInterpretable model
FaithfulnessApproximate, may diverge from the modelExact by construction
Model choiceAny model, including deep networksRestricted model families
Predictive performanceKeeps the black-box accuracyOften comparable on tabular data
EffortAdded after trainingRequires careful feature design
Audit strengthWeaker evidence of actual reasoningStronger evidence for regulators
Two routes to understandable decisions

A frequently cited argument holds that for structured, tabular problems an interpretable model often reaches similar accuracy to a black box, so explaining a black box after the fact adds risk without clear benefit. For images, audio and language, where interpretable alternatives are weak, post-hoc methods remain the practical option.

Properties that make an explanation trustworthy

A useful post-hoc explanation must be judged on several distinct quality properties, not on whether it looks plausible. An attractive heat map or a tidy ranking of features can still describe something other than what the model actually computes.

PropertyQuestion it answersHow it is tested
FidelityDoes it reflect the model’s real behaviorRemove top features and measure the output drop
StabilityDo similar inputs get similar explanationsRepeat on small perturbations
Sensitivity to the modelDoes it change when the model changesRandomize weights and compare
ComprehensibilityCan the intended audience understand itUser studies with real decision makers
ActionabilityCan the person act on itCheck that suggested changes are feasible
Quality criteria for explanations

Sanity checks are important here. Research on saliency maps showed that some popular methods produced almost the same images when the model’s weights were randomized, which means those maps reflected the input image rather than learned reasoning. Any explanation method used in a governance process should pass such randomization tests before its outputs are trusted.

Role in fairness, accountability and regulation

Post-hoc explanations support AI accountability by giving auditors, regulators and affected people a view into individual decisions. In an AI audit, global attributions can reveal that a model relies heavily on a proxy for a protected attribute, such as postal code standing in for ethnicity, which is a sign of possible proxy discrimination.

Legal frameworks increase the demand for explanations. The EU General Data Protection Regulation gives people rights around solely automated decisions with significant effects, the EU AI Act sets transparency duties for high-risk systems, and United States credit law requires adverse action notices that state principal reasons for a denial. Post-hoc attributions are often used to generate such reasons, which makes their accuracy a compliance issue as well as a technical one.

Explanations also enable contestability: a counterfactual statement such as “the application would have been approved with an income 4,000 dollars higher” tells the person what was decisive and gives a concrete basis for appeal.

Limitations and misuse

Post-hoc explanation has well-documented failure modes that practitioners need to anticipate before relying on it in sensitive settings.

  • Disagreement between methods: SHAP, LIME and gradient methods often rank features differently for the same prediction, and there is no automatic way to declare one correct.
  • Manipulation risk: studies have shown that a model can be built to hide discriminatory behavior from perturbation-based explainers, producing fairwashing, an innocent-looking explanation for a biased system.
  • Correlated features: attribution methods split credit unpredictably among correlated inputs, which can hide the real driver.
  • Causal misreading: an attribution shows what the model used, not what causes the outcome in the real world.
  • Automation bias: a confident explanation can make reviewers trust a wrong prediction more, not less.

For these reasons, an explanation should be treated as evidence to investigate, not as proof that a model is fair or correct. Pairing several methods, documenting their settings and validating them against known behavior reduces, but does not remove, these risks.

The idea in brief

Many AI systems make decisions in ways that even their builders cannot read directly. A post-hoc explanation is a description added afterward that tries to show why the system decided as it did, for example which pieces of information mattered most.

It is a bit like asking a witness to describe what happened: the account is helpful, but it is not the event itself and can leave things out. Good practice is to check such explanations carefully before trusting them.

A practical scenario

A regional bank uses a gradient-boosted model to score small-business loan applications. Before deployment, its model risk team computes SHAP values for a sample of 5,000 past decisions. The global summary shows that cash-flow ratios and payment history dominate, as expected, but the applicant’s postal code also carries noticeable weight.

The team investigates and finds that postal code correlates with neighborhoods historically denied credit. They remove the feature and retrain, accept a small drop in accuracy, and run a disparate impact analysis on the new model. For individual denials, the bank now produces adverse action reasons from the top negative attributions and adds a counterfactual statement that tells each applicant which change would most improve the outcome. Reviewers check a sample of these explanations every quarter for stability.