Feature importance

On this page 7

Feature importance is a numerical score that estimates how much each input variable contributes to the predictions or the predictive accuracy of a trained model. In machine learning, it is one of the most widely used tools for model interpretation, helping practitioners see which signals a model relies on, detect data problems and communicate results to stakeholders in artificial intelligence projects.

Feature importance is not a single method but a family of techniques with different definitions of what “important” means. Some measure how much a variable improves the model’s internal fitting criterion, others measure how much performance drops when a variable is disrupted, and others attribute each individual prediction to its inputs. Choosing the right definition matters, because the methods can rank the same features very differently.

How feature importance scores are computed

Feature importance scores are computed either from the internal structure of the model or by probing the trained model from outside. The first group is called model-specific importance, while the second is model-agnostic and works with any predictor that accepts inputs and returns outputs.

In a decision tree, random forest or gradient boosting ensemble, the classic score is mean decrease in impurity (MDI). Every split reduces an impurity measure such as Gini impurity or variance, and the reduction is credited to the feature used for that split. Summed across all trees and normalized, these reductions give a gain-based importance. Linear models offer a simpler route: the magnitude of a standardized coefficient shows how strongly the prediction changes per standard deviation of the input.

Permutation importance takes the model-agnostic route. It randomly shuffles one feature column in a held-out dataset, breaking its relationship with the target, and measures the resulting loss in a chosen metric such as accuracy, AUC or mean squared error. A large drop means the model depends heavily on that feature. The shuffle is usually repeated 5 to 30 times to estimate the variability of the score.

The following table summarizes the main computation approaches and what each score actually quantifies.

MethodWhat it measuresModel scopeData used
Mean decrease in impurityTotal impurity reduction from splitsTree-based modelsTraining data
Split countNumber of times a feature is used to splitTree-based modelsTraining data
Standardized coefficientsChange in output per standard deviationLinear and logistic modelsTraining data
Permutation importanceMetric loss after shuffling a featureAny modelValidation or test data
Drop-column importanceMetric loss after retraining without a featureAny modelValidation data, many retrains
SHAP valuesAverage contribution to each predictionAny model, fast for treesAny dataset
Main methods for computing feature importance

Global and local feature importance

Feature importance can describe the model as a whole (global importance) or one specific prediction (local importance). Global scores answer which features matter on average across a dataset, while local scores explain why a particular customer, transaction or image received its specific output.

SHAP (SHapley Additive exPlanations) bridges both levels. It applies Shapley values from cooperative game theory, distributing the difference between a prediction and the average prediction fairly among the features. Averaging the absolute SHAP values over many rows yields a global ranking, while individual values explain single predictions. TreeSHAP computes exact values efficiently for tree ensembles, whereas KernelSHAP approximates them for arbitrary models at a much higher computational cost.

The table below contrasts the two levels of explanation and their typical consumers.

AspectGlobal importanceLocal importance
Question answeredWhich features drive the model overallWhy this prediction has this value
Typical methodsMDI, permutation, mean absolute SHAPSHAP values, LIME, counterfactuals
Typical audienceData scientists, model reviewersCase handlers, affected individuals, auditors
Main useFeature selection, model debuggingDecision justification, error analysis
Global versus local importance

How feature importance relates to feature selection

Feature importance is an input to feature selection, not a synonym for it. Feature selection is the decision to keep or remove variables, and importance scores are one of several signals that guide that decision, alongside correlation analysis, domain knowledge and regularization.

Regularized models such as lasso regression and elastic net perform selection implicitly by driving some coefficients to exactly zero, whereas ridge regression only shrinks them. Recursive feature elimination repeatedly trains a model, removes the least important features and retrains, using importance scores as its ranking criterion. Feature extraction methods such as principal component analysis follow a different logic: they build new composite variables, so importance then refers to components rather than original inputs, which reduces interpretability.

Choosing a feature importance method

The right method depends on the question being asked, the model type and the available computing budget. A quick diagnostic during experimentation calls for different tooling than a documented explanation for a regulated credit decision.

The following decision table maps common situations to a recommended approach and the reasoning behind it.

SituationRecommended methodReason
Quick check of a tree ensembleGain-based importanceFree to compute after training
Comparing features fairly across typesPermutation importance on test dataAvoids bias toward high-cardinality features
Explaining individual decisionsSHAP valuesConsistent, additive per-prediction attribution
Strongly correlated inputsGrouped permutation or conditional methodsPlain shuffling creates unrealistic rows
Small dataset, few featuresDrop-column importanceRetraining cost is affordable and exact
Linear model with scaled inputsStandardized coefficientsDirectly interpretable and signed
Decision guide for selecting an importance method

In practice, experienced teams compare at least two methods. When gain-based and permutation rankings disagree sharply, the disagreement itself is a useful diagnostic signal that points to correlated features, overfitting or a leaking variable.

Limitations and common mistakes

Feature importance describes what the model uses, not how the world works. A high score shows predictive reliance, never causation, and a model can rely on a proxy variable that merely correlates with the true driver. Several technical pitfalls make the scores misleading even as a description of the model.

MistakeConsequenceCorrection
Trusting impurity scores aloneInflated scores for high-cardinality or noisy featuresConfirm with permutation importance on held-out data
Computing permutation on training dataOverfitted features appear importantUse a validation or test set
Ignoring correlated featuresImportance is split or hidden between twinsGroup correlated features before scoring
Reading importance as causal effectWrong business or policy conclusionsUse causal inference methods for effects
Ranking from a single runUnstable order for close scoresRepeat with several seeds and report intervals
Frequent errors and how to correct them

Data leakage often reveals itself through importance analysis. When one feature dominates with an implausibly large score, such as a timestamp or an identifier that encodes the outcome, the model has usually learned a shortcut that will fail in production. Importance scores also drift over time as data distributions change, so they should be recomputed after retraining and monitored alongside concept drift.

Put simply

Feature importance tells you which pieces of information a model pays attention to. Imagine a teacher grading essays who looks at spelling, length and argument quality. If you hid the argument quality and the grades changed sharply, you would know the teacher cares about it most.

Feature importance does the same with a computer model. It hides or scrambles one input at a time and watches how much the results suffer. The key point is that it shows what the model relies on, not what causes the outcome in real life.

In practice

A data science team at a regional bank builds a gradient boosting model to predict loan defaults from 40 applicant features. The built-in gain importance ranks an internal application ID field among the top three, which makes no business sense. Permutation importance on a held-out test set confirms the dependence: shuffling that field lowers AUC from 0.81 to 0.72.

Investigation shows that application IDs were assigned in batches, and one batch came from a period with unusually high defaults. The team removes the field, retrains and recomputes importance. Debt-to-income ratio, payment history and loan term now lead the ranking, test AUC settles at a realistic 0.78, and SHAP values provide per-applicant explanations for the credit review team. The outcome is a slightly less accurate but trustworthy model that no longer depends on a leaking identifier.