Feature selection

On this page 8

Feature selection is the process of choosing a subset of the original input variables that carries the most predictive information for a model, while discarding variables that are irrelevant or redundant. In machine learning, it differs from feature extraction because it keeps the selected columns unchanged instead of transforming them into new composite variables, so the resulting model stays interpretable in terms of the original measurements.

Practitioners select features to reduce overfitting, shorten training and inference time, lower data collection costs and make models easier to explain. The task is hard because the number of possible subsets grows as 2 to the power of p for p candidate features, so every practical method is a heuristic that trades search effort against the quality of the chosen subset.

How feature selection works

Feature selection works by scoring candidate features or subsets against a criterion and keeping the ones that score best. The criterion can be a statistical measure of relevance, such as correlation or mutual information with the target, or the validated performance of a model trained on the subset.

Three ideas guide every method. A feature is relevant when it carries information about the target, redundant when another selected feature already carries the same information, and irrelevant when it carries none. A good subset is maximally relevant and minimally redundant, a principle formalized in the mRMR criterion (minimum redundancy, maximum relevance).

Relevance is not always visible one feature at a time. In the classic XOR pattern, two features are individually useless but jointly decisive, which is why univariate screening can discard variables that a multivariate search would keep.

Filter, wrapper and embedded methods

Methods fall into three families that differ in whether and how they consult a learning algorithm. The table below compares them on the properties that usually decide the choice.

FamilyHow it scores featuresTypical techniquesCostMain weakness
Filter methodsStatistic computed independently of any modelPearson correlation, chi-squared, ANOVA F-test, mutual informationLowIgnores feature interactions and the final model
Wrapper methodsCross-validated performance of a model on each subsetForward selection, backward elimination, recursive feature eliminationHighExpensive and prone to overfitting the selection
Embedded methodsSelection happens during model trainingLasso regression, elastic net, tree-based importanceMediumSelection is tied to one model family
Main families of feature selection methods

Recursive feature elimination (RFE) is the best-known wrapper: it trains a model, removes the least important features, and repeats until a target size is reached. Lasso regression is the best-known embedded method, because its L1 penalty drives coefficients exactly to zero and so removes features as a side effect of regularization.

Common scoring criteria and when they apply

The scoring criterion must match the data types of the feature and the target, otherwise the scores are meaningless. The following table lists widely used criteria and their valid settings.

CriterionFeature typeTarget typeWhat it detects
Pearson correlationNumericNumericLinear association only
ANOVA F-testNumericCategoricalDifferences in class means
Chi-squared testCategorical or countsCategoricalDependence between frequencies
Mutual informationAnyAnyLinear and nonlinear dependence
Variance thresholdAnyNot usedNear-constant features
Permutation importanceAnyAnyLoss of model accuracy when shuffled
Scoring criteria by data type

Mutual information is the most general choice, but its estimates are noisy on small samples and depend on the estimator, such as binning or the k-nearest neighbors estimator. Permutation importance is model-agnostic and reflects actual predictive value, yet it can understate correlated features because the model falls back on their substitutes.

Feature selection is one form of dimensionality reduction, and it is often confused with neighboring techniques in the same pipeline. The comparison below separates them by what they produce.

TechniqueOutputKeeps original variablesInterpretability
Feature selectionSubset of existing columnsYesHigh
Feature extractionNew derived variablesNoModerate to low
Principal component analysisLinear combinations of all inputsNoLow
RegularizationShrunk or zeroed coefficientsYes, when L1 is usedHigh for sparse models
Feature importanceRanking of featuresYesDepends on method
How feature selection relates to neighboring techniques

A feature importance ranking is not yet a selection: it becomes one only when a cutoff is chosen and validated. Principal component analysis, by contrast, still requires every original input at prediction time, so it does not reduce data collection costs.

Choosing a method in practice

The right method depends on the ratio of features to samples, the compute budget and whether interpretability matters. The decision table below summarizes typical choices.

SituationRecommended approachReason
Thousands of features, few samplesFilter screening, then LassoWrappers would overfit and cost too much
Tens of features, ample dataRecursive feature eliminationSearch is affordable and model-aware
Strongly correlated groupsElastic net or group-wise selectionLasso picks one member arbitrarily
Tree ensembles as final modelPermutation importance with validationImpurity importance favors high-cardinality features
Regulated or audited modelsEmbedded sparse linear modelCoefficients stay directly explainable
Decision guide for selecting features

Whatever the method, the number of retained features should be treated as a hyperparameter tuned by cross-validation, not fixed by intuition. Stability can be checked by repeating the selection on bootstrap samples and keeping features that are chosen in, for example, more than 70 to 80 percent of runs.

Pitfalls and misconceptions

The most damaging mistake is selection leakage: choosing features on the full dataset and only then cross-validating the model. Because the test folds already influenced the choice, the reported accuracy is optimistically biased, sometimes dramatically on high-dimensional data.

  • Selection must run inside each fold of cross-validation, as a step of the training pipeline.
  • Dropping a feature is not proof of irrelevance: it may be redundant with a kept feature or useful only jointly with others.
  • Unstable subsets across resamples signal that the data cannot distinguish between correlated candidates.
  • Selection does not establish causality: a predictive feature may be a proxy or a consequence of the target.
  • Concept drift can make a once-useful subset obsolete, so selections need periodic review in production.

Modern gradient boosting and random forest models tolerate many weak features, so aggressive selection may bring little accuracy gain for them. The benefit then lies mainly in lower cost, faster inference and simpler monitoring.

A non-technical view

Imagine packing for a trip with a small suitcase. You lay out everything you own and keep only the items you will actually use. Feature selection does the same for a prediction model: from many possible pieces of information, it keeps the ones that genuinely help the prediction and leaves the rest behind.

Carrying less makes the trip easier. A model with fewer inputs is faster, cheaper and easier to explain, and it is less likely to be fooled by coincidences that happen to appear in old data but never repeat.

A concrete example

A hospital analytics team builds a model to predict 30-day readmission of heart failure patients. The electronic health records offer about 400 candidate variables, from laboratory values to billing codes, for roughly 12,000 past admissions.

The team first removes near-constant variables, then runs an L1-regularized logistic regression inside a five-fold cross-validation pipeline, so selection never sees the validation patients. Variables chosen in fewer than 80 percent of bootstrap runs are dropped. The final model uses 28 variables, including prior admissions, serum sodium and discharge medications.

Validated accuracy matches the full 400-variable model within measurement noise, but clinicians can now review every input. The hospital also stops collecting several costly variables for the model, and the shorter input list makes data quality monitoring practical.