Integrated gradients

On this page 7

Integrated gradients is a feature attribution method for differentiable models that assigns each input feature a share of a prediction by accumulating gradients along a straight path from a neutral baseline input to the actual input. Introduced in 2017 by Sundararajan, Taly and Yan, it is one of the most widely used post-hoc explanation techniques for deep neural networks.

In AI ethics, integrated gradients supports transparency and accountability: auditors use it to see which words, pixels or tabular fields drove a decision and to detect when a model relies on protected attributes or their proxies. Its appeal comes from a small set of formal axioms that simpler gradient methods violate.

How integrated gradients computes attributions

Integrated gradients attributes a prediction by averaging the model gradient over many points between a baseline x’ and the input x, then multiplying by the difference between the two. For feature i, the attribution equals (x_i minus x’_i) times the integral of the partial derivative of the model output along the path α from 0 to 1.

In practice the integral is approximated with a Riemann sum. The input is scaled in m steps, the gradient is computed at each step, and the gradients are averaged. Each explanation therefore costs about m forward and backward passes, which can be batched on a GPU.

The table below lists the components of a typical computation and the values practitioners use.

ComponentRoleTypical setting
BaselineReference input meaning absence of signalBlack image, zero embedding, data mean
PathRoute from baseline to inputStraight line in input space
Number of stepsResolution of the Riemann approximation20 to 300 steps
Target outputQuantity being explainedLogit or probability of one class
Completeness gapSum of attributions minus output differenceBelow about 5 percent of the difference
Components and typical settings

The completeness gap doubles as a convergence check. If the attributions do not add up closely to the difference between the output at the input and at the baseline, the step count is too low or the baseline is poorly chosen.

The axioms behind the method

Integrated gradients is defined by axioms that make its attributions predictable and comparable. The original paper argued that an attribution method should satisfy them, and showed that plain gradients and several popular alternatives do not.

PropertyMeaningWhy it matters for audits
SensitivityA feature that changes the output receives nonzero attributionInfluential features cannot hide behind saturation
Implementation invarianceFunctionally identical networks get identical attributionsResults depend on behavior, not architecture details
CompletenessAttributions sum to output minus baseline outputEvery unit of the score is accounted for
LinearityAttributions of combined models combine linearlyEnsembles can be explained consistently
Symmetry preservationSymmetric features with equal values get equal creditNo arbitrary favoritism among equivalent inputs
Axioms satisfied by integrated gradients

Integrated gradients is a special case of the Aumann-Shapley value from cost-sharing theory, which links it to the game-theoretic foundations of Shapley-based methods. It is, however, a path method with one fixed path, so it distributes credit for feature interactions differently than exact Shapley values do.

Integrated gradients compared with other explanation methods

Integrated gradients occupies a middle ground between cheap gradient maps and expensive perturbation methods. It needs model gradients, so it is a white-box technique, but it avoids the saturation problem of single gradients.

MethodModel accessMain strengthMain weakness
Integrated gradientsGradients requiredAxiomatic, completeness holdsDepends on baseline choice
Saliency mapGradients requiredOne backward passSaturation hides important features
Grad-CAMConvolutional layer activationsCoarse, readable region mapsLow resolution, image models only
SHAPBlack-box or model-specificGame-theoretic fairness of creditCostly, needs background data
LIMEBlack-boxModel-agnostic local surrogateUnstable across random samples
Comparison with neighboring attribution methods

Variants address specific weaknesses. Expected gradients averages over many baselines drawn from the data, blur integrated gradients uses progressively blurred images, and adding SmoothGrad noise reduces visual speckle in image attributions.

Choosing a baseline and applying the method

The baseline determines what question integrated gradients answers: attributions explain the prediction relative to the baseline, not in absolute terms. A baseline should produce a near-neutral prediction and represent a meaningful absence of information for the domain.

Data typeCommon baselineCaveat
ImagesAll-black or blurred imageBlack pixels in the input receive zero attribution
TextPadding or zero token embeddingsZero embedding may lie off the data manifold
Tabular dataFeature means or mediansMean values may be unrealistic combinations
Any modalitySeveral sampled baselines averagedHigher cost, smoother results
Baseline choices by data type

For fairness work, auditors compute attributions for many individuals and aggregate them by group. Consistently high attribution to a ZIP code, a first name or a gap in employment history can reveal proxy discrimination even when the protected attribute itself was excluded from training.

Limitations and common misinterpretations

Integrated gradients explains model sensitivity, not causation in the world, and its outputs are easy to over-read. Several limitations recur in practice.

  • Baseline dependence: different reasonable baselines can produce visibly different attribution maps.
  • Off-manifold paths: intermediate points on the straight path may be unrealistic inputs the model never saw.
  • Interaction blindness: a single attribution per feature cannot show that two features matter only together.
  • No ground truth: plausible-looking maps can still be unfaithful, so sanity checks such as model randomization tests are advisable.
  • Manipulability: models can be trained adversarially to produce misleading explanations, which limits use as sole audit evidence.

Responsible use pairs integrated gradients with counterfactual tests and fairness metrics rather than treating a heat map as proof that a model is fair or unfair.

In plain words

Integrated gradients answers a simple question: which parts of the input pushed the model toward its answer. It starts from an empty version of the input, such as a black picture, and slowly fades in the real input while watching how the model reacts.

Every feature collects credit for the change it caused along the way. At the end, the credits add up to the full difference between the empty answer and the real one. It is like tracing which ingredients changed the taste as a recipe is gradually completed.

A real-world example

A bank’s model risk team reviews a neural network that scores loan applications. Gender and ethnicity were removed from the training data, but the team must show regulators that the model does not discriminate indirectly.

The team computes integrated gradients for 20,000 recent applications with 64 steps and a baseline of median feature values, then averages the attributions by demographic group using data held separately for auditing. The completeness gap stays below 2 percent, which confirms the approximation is adequate.

The analysis shows that postal code carries unusually high negative attribution for applicants from two neighborhoods with mostly minority residents. The team replaces the raw postal code with a coarser regional income feature, retrains the model and confirms with demographic parity checks that the approval gap narrows from 9 to 3 percentage points.