Integrated gradients is a feature attribution method for differentiable models that assigns each input feature a share of a prediction by accumulating gradients along a straight path from a neutral baseline input to the actual input. Introduced in 2017 by Sundararajan, Taly and Yan, it is one of the most widely used post-hoc explanation techniques for deep neural networks.
In AI ethics, integrated gradients supports transparency and accountability: auditors use it to see which words, pixels or tabular fields drove a decision and to detect when a model relies on protected attributes or their proxies. Its appeal comes from a small set of formal axioms that simpler gradient methods violate.
How integrated gradients computes attributions
Integrated gradients attributes a prediction by averaging the model gradient over many points between a baseline x’ and the input x, then multiplying by the difference between the two. For feature i, the attribution equals (x_i minus x’_i) times the integral of the partial derivative of the model output along the path α from 0 to 1.
In practice the integral is approximated with a Riemann sum. The input is scaled in m steps, the gradient is computed at each step, and the gradients are averaged. Each explanation therefore costs about m forward and backward passes, which can be batched on a GPU.
The table below lists the components of a typical computation and the values practitioners use.
| Component | Role | Typical setting |
|---|---|---|
| Baseline | Reference input meaning absence of signal | Black image, zero embedding, data mean |
| Path | Route from baseline to input | Straight line in input space |
| Number of steps | Resolution of the Riemann approximation | 20 to 300 steps |
| Target output | Quantity being explained | Logit or probability of one class |
| Completeness gap | Sum of attributions minus output difference | Below about 5 percent of the difference |
The completeness gap doubles as a convergence check. If the attributions do not add up closely to the difference between the output at the input and at the baseline, the step count is too low or the baseline is poorly chosen.
The axioms behind the method
Integrated gradients is defined by axioms that make its attributions predictable and comparable. The original paper argued that an attribution method should satisfy them, and showed that plain gradients and several popular alternatives do not.
| Property | Meaning | Why it matters for audits |
|---|---|---|
| Sensitivity | A feature that changes the output receives nonzero attribution | Influential features cannot hide behind saturation |
| Implementation invariance | Functionally identical networks get identical attributions | Results depend on behavior, not architecture details |
| Completeness | Attributions sum to output minus baseline output | Every unit of the score is accounted for |
| Linearity | Attributions of combined models combine linearly | Ensembles can be explained consistently |
| Symmetry preservation | Symmetric features with equal values get equal credit | No arbitrary favoritism among equivalent inputs |
Integrated gradients is a special case of the Aumann-Shapley value from cost-sharing theory, which links it to the game-theoretic foundations of Shapley-based methods. It is, however, a path method with one fixed path, so it distributes credit for feature interactions differently than exact Shapley values do.
Integrated gradients compared with other explanation methods
Integrated gradients occupies a middle ground between cheap gradient maps and expensive perturbation methods. It needs model gradients, so it is a white-box technique, but it avoids the saturation problem of single gradients.
| Method | Model access | Main strength | Main weakness |
|---|---|---|---|
| Integrated gradients | Gradients required | Axiomatic, completeness holds | Depends on baseline choice |
| Saliency map | Gradients required | One backward pass | Saturation hides important features |
| Grad-CAM | Convolutional layer activations | Coarse, readable region maps | Low resolution, image models only |
| SHAP | Black-box or model-specific | Game-theoretic fairness of credit | Costly, needs background data |
| LIME | Black-box | Model-agnostic local surrogate | Unstable across random samples |
Variants address specific weaknesses. Expected gradients averages over many baselines drawn from the data, blur integrated gradients uses progressively blurred images, and adding SmoothGrad noise reduces visual speckle in image attributions.
Choosing a baseline and applying the method
The baseline determines what question integrated gradients answers: attributions explain the prediction relative to the baseline, not in absolute terms. A baseline should produce a near-neutral prediction and represent a meaningful absence of information for the domain.
| Data type | Common baseline | Caveat |
|---|---|---|
| Images | All-black or blurred image | Black pixels in the input receive zero attribution |
| Text | Padding or zero token embeddings | Zero embedding may lie off the data manifold |
| Tabular data | Feature means or medians | Mean values may be unrealistic combinations |
| Any modality | Several sampled baselines averaged | Higher cost, smoother results |
For fairness work, auditors compute attributions for many individuals and aggregate them by group. Consistently high attribution to a ZIP code, a first name or a gap in employment history can reveal proxy discrimination even when the protected attribute itself was excluded from training.
Limitations and common misinterpretations
Integrated gradients explains model sensitivity, not causation in the world, and its outputs are easy to over-read. Several limitations recur in practice.
- Baseline dependence: different reasonable baselines can produce visibly different attribution maps.
- Off-manifold paths: intermediate points on the straight path may be unrealistic inputs the model never saw.
- Interaction blindness: a single attribution per feature cannot show that two features matter only together.
- No ground truth: plausible-looking maps can still be unfaithful, so sanity checks such as model randomization tests are advisable.
- Manipulability: models can be trained adversarially to produce misleading explanations, which limits use as sole audit evidence.
Responsible use pairs integrated gradients with counterfactual tests and fairness metrics rather than treating a heat map as proof that a model is fair or unfair.
In plain words
Integrated gradients answers a simple question: which parts of the input pushed the model toward its answer. It starts from an empty version of the input, such as a black picture, and slowly fades in the real input while watching how the model reacts.
Every feature collects credit for the change it caused along the way. At the end, the credits add up to the full difference between the empty answer and the real one. It is like tracing which ingredients changed the taste as a recipe is gradually completed.
A real-world example
A bank’s model risk team reviews a neural network that scores loan applications. Gender and ethnicity were removed from the training data, but the team must show regulators that the model does not discriminate indirectly.
The team computes integrated gradients for 20,000 recent applications with 64 steps and a baseline of median feature values, then averages the attributions by demographic group using data held separately for auditing. The completeness gap stays below 2 percent, which confirms the approximation is adequate.
The analysis shows that postal code carries unusually high negative attribution for applicants from two neighborhoods with mostly minority residents. The team replaces the raw postal code with a coarser regional income feature, retrains the model and confirms with demographic parity checks that the approval gap narrows from 9 to 3 percentage points.