Saliency map

On this page 7

A saliency map is a visual explanation that assigns an importance score to each input feature, most often each pixel of an image, indicating how strongly that feature influenced a machine learning model’s prediction. In AI ethics, saliency maps are one of the most widely used tools for AI transparency, because they let auditors, developers and affected people see where a model was looking when it made a decision.

Saliency maps belong to the family of post-hoc explanations: they are computed after a model is trained and do not change how it works. Their appeal is intuitive, since a colored overlay is easy to read, but research has shown that a convincing heatmap is not always a faithful explanation, which makes careful interpretation essential.

How a saliency map is computed

A saliency map is computed by asking how the model’s output would change if each input feature changed slightly. The classic method, often called vanilla gradient saliency, takes the gradient of the class score with respect to the input pixels and displays its absolute magnitude as a heatmap.

Because raw gradients are noisy, later methods refine the idea. Some average gradients over many noisy copies of the input, some integrate gradients along a path from a neutral baseline, and others use the activations of the last convolutional layer to produce coarser but cleaner maps. A separate group of perturbation methods hides parts of the image and measures how much the prediction drops.

The table below lists the main saliency techniques and how they differ.

MethodBasic principleModel access neededTypical output
Vanilla gradientInput gradient of class scoreGradientsFine-grained, noisy
SmoothGradAverages gradients over noisy inputsGradientsSharper, less speckled
Integrated gradientsAccumulates gradients from a baselineGradientsSatisfies completeness axiom
Grad-CAMWeights final feature maps by gradientsInternal activationsCoarse class regions
OcclusionMasks patches and measures score dropPredictions onlyBlocky, model-agnostic
RISERandom masks weighted by outputPredictions onlySmooth, slow to compute
Common saliency map techniques

Why saliency maps matter in AI ethics

Saliency maps matter in AI ethics because they can expose shortcut learning, where a model reaches correct answers for the wrong reasons. A well-known pattern is a medical imaging classifier that attends to hospital markers or scanner labels rather than to the pathology, or a wildlife classifier that relies on snowy backgrounds instead of the animal.

Such findings support several ethical goals at once. They inform AI audit work, provide evidence for an algorithmic impact assessment, help detect proxy discrimination when a model focuses on features correlated with a protected attribute, and give organizations material for AI accountability and contestability processes.

Ethical goalHow the saliency map helpsLimit of that help
Bias detectionShows attention on irrelevant or sensitive regionsDoes not prove the absence of bias
Model debuggingReveals spurious correlations in training dataRequires expert interpretation
User trustGives a visible reason for a decisionCan create false confidence
Regulatory transparencyDocuments model behavior for reviewersRarely sufficient on its own
Ethical uses of saliency maps

Saliency maps compared with other explanation methods

Saliency maps differ from other explanation methods mainly in their output format and granularity. They produce a dense, per-feature heatmap for one input, whereas SHAP and LIME usually summarize attributions over interpretable features, and a counterfactual explanation describes the smallest change that would flip the decision.

ApproachExplanation formBest suited toMain caution
Saliency mapPixel or token heatmapImages, audio, textFaithfulness varies by method
SHAPAdditive feature contributionsTabular dataCostly for large inputs
LIMELocal surrogate model weightsAny data typeUnstable across runs
Counterfactual explanationMinimal change to alter outcomeIndividual decisionsMay suggest unrealistic changes
Interpretable modelTransparent model structureHigh-stakes tabular tasksPossible accuracy cost
Saliency maps and related explanation approaches

Saliency maps also extend beyond vision. In language models they highlight influential tokens, and in audio they mark time-frequency regions, although the visual clarity that makes image heatmaps persuasive is often weaker for these data types.

Faithfulness and known failure modes

The central problem with saliency maps is faithfulness: whether the highlighted regions truly reflect the model’s reasoning. Sanity-check experiments showed that several popular methods produce nearly identical heatmaps for trained and randomized models, meaning they partly reflect image edges rather than learned behavior.

  • Confirmation bias: viewers tend to accept maps that match their expectations and dismiss those that do not.
  • Adversarial manipulation: models can be modified so their saliency maps hide reliance on a sensitive feature, a form of fairwashing.
  • Baseline sensitivity: integrated gradients change considerably with the choice of black, blurred or random baseline.
  • Missing interactions: a heatmap shows where, not how features combine, so it cannot explain conditional logic.
  • Resolution limits: coarse methods such as Grad-CAM may blur small but decisive details.

Practitioners therefore recommend quantitative evaluation alongside visual inspection, for example deletion and insertion curves that measure how quickly the prediction changes as the most salient pixels are removed or restored.

Using saliency maps responsibly

Saliency maps are most useful when treated as evidence to investigate, not as final proof of fairness or correctness. Responsible teams compare several methods, run sanity checks and confirm suspicious patterns with controlled experiments on the data.

SituationRecommended practiceReason
Auditing an image classifierCompare at least two methodsAgreement increases confidence
Explaining a decision to usersPair map with plain-language textHeatmaps are easy to misread
Suspected shortcut learningTest with edited or counterfactual imagesConfirms a causal effect
Regulated high-stakes useDocument method, baseline and limitsSupports accountability and review
Practical guidance for saliency analysis

In everyday language

A saliency map is like a highlighter pen for a computer’s attention. When an AI system looks at a photo and decides what it shows, the map colors the parts of the picture that mattered most for that answer.

This helps people check whether the system is focusing on the right things. The important caution is that the highlighting is itself an estimate, so it can look convincing and still be misleading. It is a useful clue, not a guarantee.

Seen in the real world

A hospital research team evaluates a chest X-ray model that flags possible pneumonia with high accuracy on internal test data. Before deployment, an ethics reviewer asks for saliency maps on a sample of 500 images from two hospitals.

Grad-CAM and integrated gradients both show strong attention on a text marker in the image corner that differs between portable and fixed scanners. Because sicker patients are more often imaged with portable devices, the model had partly learned scanner type as a shortcut for illness rather than lung findings.

The team crops and masks the markers, retrains the model and repeats the analysis. Accuracy falls slightly, but attention now concentrates on the lungs and performance becomes consistent across both hospitals. The saliency analysis did not certify the model, yet it prevented a biased system from reaching patients.