Feature selection is the process of choosing a subset of the original input variables that carries the most predictive information for a model, while discarding variables that are irrelevant or redundant. In machine learning, it differs from feature extraction because it keeps the selected columns unchanged instead of transforming them into new composite variables, so the resulting model stays interpretable in terms of the original measurements.
Practitioners select features to reduce overfitting, shorten training and inference time, lower data collection costs and make models easier to explain. The task is hard because the number of possible subsets grows as 2 to the power of p for p candidate features, so every practical method is a heuristic that trades search effort against the quality of the chosen subset.
How feature selection works
Feature selection works by scoring candidate features or subsets against a criterion and keeping the ones that score best. The criterion can be a statistical measure of relevance, such as correlation or mutual information with the target, or the validated performance of a model trained on the subset.
Three ideas guide every method. A feature is relevant when it carries information about the target, redundant when another selected feature already carries the same information, and irrelevant when it carries none. A good subset is maximally relevant and minimally redundant, a principle formalized in the mRMR criterion (minimum redundancy, maximum relevance).
Relevance is not always visible one feature at a time. In the classic XOR pattern, two features are individually useless but jointly decisive, which is why univariate screening can discard variables that a multivariate search would keep.
Filter, wrapper and embedded methods
Methods fall into three families that differ in whether and how they consult a learning algorithm. The table below compares them on the properties that usually decide the choice.
| Family | How it scores features | Typical techniques | Cost | Main weakness |
|---|---|---|---|---|
| Filter methods | Statistic computed independently of any model | Pearson correlation, chi-squared, ANOVA F-test, mutual information | Low | Ignores feature interactions and the final model |
| Wrapper methods | Cross-validated performance of a model on each subset | Forward selection, backward elimination, recursive feature elimination | High | Expensive and prone to overfitting the selection |
| Embedded methods | Selection happens during model training | Lasso regression, elastic net, tree-based importance | Medium | Selection is tied to one model family |
Recursive feature elimination (RFE) is the best-known wrapper: it trains a model, removes the least important features, and repeats until a target size is reached. Lasso regression is the best-known embedded method, because its L1 penalty drives coefficients exactly to zero and so removes features as a side effect of regularization.
Common scoring criteria and when they apply
The scoring criterion must match the data types of the feature and the target, otherwise the scores are meaningless. The following table lists widely used criteria and their valid settings.
| Criterion | Feature type | Target type | What it detects |
|---|---|---|---|
| Pearson correlation | Numeric | Numeric | Linear association only |
| ANOVA F-test | Numeric | Categorical | Differences in class means |
| Chi-squared test | Categorical or counts | Categorical | Dependence between frequencies |
| Mutual information | Any | Any | Linear and nonlinear dependence |
| Variance threshold | Any | Not used | Near-constant features |
| Permutation importance | Any | Any | Loss of model accuracy when shuffled |
Mutual information is the most general choice, but its estimates are noisy on small samples and depend on the estimator, such as binning or the k-nearest neighbors estimator. Permutation importance is model-agnostic and reflects actual predictive value, yet it can understate correlated features because the model falls back on their substitutes.
Feature selection compared with related techniques
Feature selection is one form of dimensionality reduction, and it is often confused with neighboring techniques in the same pipeline. The comparison below separates them by what they produce.
| Technique | Output | Keeps original variables | Interpretability |
|---|---|---|---|
| Feature selection | Subset of existing columns | Yes | High |
| Feature extraction | New derived variables | No | Moderate to low |
| Principal component analysis | Linear combinations of all inputs | No | Low |
| Regularization | Shrunk or zeroed coefficients | Yes, when L1 is used | High for sparse models |
| Feature importance | Ranking of features | Yes | Depends on method |
A feature importance ranking is not yet a selection: it becomes one only when a cutoff is chosen and validated. Principal component analysis, by contrast, still requires every original input at prediction time, so it does not reduce data collection costs.
Choosing a method in practice
The right method depends on the ratio of features to samples, the compute budget and whether interpretability matters. The decision table below summarizes typical choices.
| Situation | Recommended approach | Reason |
|---|---|---|
| Thousands of features, few samples | Filter screening, then Lasso | Wrappers would overfit and cost too much |
| Tens of features, ample data | Recursive feature elimination | Search is affordable and model-aware |
| Strongly correlated groups | Elastic net or group-wise selection | Lasso picks one member arbitrarily |
| Tree ensembles as final model | Permutation importance with validation | Impurity importance favors high-cardinality features |
| Regulated or audited models | Embedded sparse linear model | Coefficients stay directly explainable |
Whatever the method, the number of retained features should be treated as a hyperparameter tuned by cross-validation, not fixed by intuition. Stability can be checked by repeating the selection on bootstrap samples and keeping features that are chosen in, for example, more than 70 to 80 percent of runs.
Pitfalls and misconceptions
The most damaging mistake is selection leakage: choosing features on the full dataset and only then cross-validating the model. Because the test folds already influenced the choice, the reported accuracy is optimistically biased, sometimes dramatically on high-dimensional data.
- Selection must run inside each fold of cross-validation, as a step of the training pipeline.
- Dropping a feature is not proof of irrelevance: it may be redundant with a kept feature or useful only jointly with others.
- Unstable subsets across resamples signal that the data cannot distinguish between correlated candidates.
- Selection does not establish causality: a predictive feature may be a proxy or a consequence of the target.
- Concept drift can make a once-useful subset obsolete, so selections need periodic review in production.
Modern gradient boosting and random forest models tolerate many weak features, so aggressive selection may bring little accuracy gain for them. The benefit then lies mainly in lower cost, faster inference and simpler monitoring.
A non-technical view
Imagine packing for a trip with a small suitcase. You lay out everything you own and keep only the items you will actually use. Feature selection does the same for a prediction model: from many possible pieces of information, it keeps the ones that genuinely help the prediction and leaves the rest behind.
Carrying less makes the trip easier. A model with fewer inputs is faster, cheaper and easier to explain, and it is less likely to be fooled by coincidences that happen to appear in old data but never repeat.
A concrete example
A hospital analytics team builds a model to predict 30-day readmission of heart failure patients. The electronic health records offer about 400 candidate variables, from laboratory values to billing codes, for roughly 12,000 past admissions.
The team first removes near-constant variables, then runs an L1-regularized logistic regression inside a five-fold cross-validation pipeline, so selection never sees the validation patients. Variables chosen in fewer than 80 percent of bootstrap runs are dropped. The final model uses 28 variables, including prior admissions, serum sodium and discharge medications.
Validated accuracy matches the full 400-variable model within measurement noise, but clinicians can now review every input. The hospital also stops collecting several costly variables for the model, and the shorter input list makes data quality monitoring practical.