Dimensionality reduction is the family of machine learning techniques that transform data with many features into a representation with far fewer dimensions while preserving the structure that matters for a given task, such as variance, distances, class separation or neighborhood relations. In artificial intelligence practice, it serves three distinct purposes: compressing inputs for faster and more stable models, visualizing high-dimensional data in two or three dimensions, and removing noise and redundancy before clustering or classification.
The method answers a practical problem known as the curse of dimensionality: as the number of features grows, data points become sparse, distances lose contrast and models need exponentially more samples to generalize. Reducing dimensions trades some information for better statistical efficiency, lower computational cost and clearer interpretation.
How dimensionality reduction works
Every dimensionality reduction method learns a mapping from the original feature space to a lower-dimensional space, guided by an objective that defines what must survive the compression. Real data rarely fills its full feature space; it tends to lie near a low-dimensional manifold, and the methods try to recover the coordinates of that manifold.
There are two broad strategies. Feature selection keeps a subset of the original columns and discards the rest, so the retained features stay interpretable. Feature extraction builds new variables as combinations or nonlinear functions of all inputs, which usually preserves more information but produces components without a direct physical meaning.
The table below summarizes the main objectives that different methods optimize.
| Objective | What is preserved | Typical method |
|---|---|---|
| Maximum variance | Directions with the largest spread | Principal component analysis |
| Class separation | Ratio of between-class to within-class scatter | Linear discriminant analysis |
| Local neighborhoods | Which points are near each other | t-SNE, UMAP |
| Reconstruction accuracy | Ability to rebuild the input | Autoencoder |
| Pairwise distances | Global distance structure | Multidimensional scaling, random projection |
Linear and nonlinear methods
Linear methods project data onto a hyperplane spanned by a few basis vectors, which makes them fast, deterministic and reversible up to the discarded components. Principal component analysis (PCA) is the reference case: it computes the eigenvectors of the covariance matrix, usually through singular value decomposition, and orders them by the variance they explain.
Nonlinear methods handle data that lies on curved manifolds, where a straight projection would fold distant points on top of each other. Kernel PCA applies the kernel trick, Isomap preserves geodesic distances along a neighborhood graph, and t-SNE and UMAP optimize low-dimensional embeddings so that local neighbors stay close. Neural autoencoders learn an encoder and decoder jointly and scale to images, audio and text.
The following comparison helps choose among the most frequently used techniques.
| Method | Type | Supervision | New data | Main strength |
|---|---|---|---|---|
| PCA | Linear | Unsupervised | Direct projection | Fast, interpretable variance ranking |
| LDA | Linear | Supervised | Direct projection | At most C minus 1 discriminative axes |
| Kernel PCA | Nonlinear | Unsupervised | Kernel evaluation | Captures curved structure |
| t-SNE | Nonlinear | Unsupervised | No native transform | Sharp local cluster views |
| UMAP | Nonlinear | Optional | Approximate transform | Faster, better global layout |
| Autoencoder | Nonlinear | Self-supervised | Encoder pass | Scales to raw signals |
Choosing the target number of dimensions
The number of output dimensions is a hyperparameter that depends on the downstream goal, not a property of the data alone. For visualization, the answer is fixed at two or three dimensions. For modeling, the dimension should be selected by validation performance or by an explicit information budget.
With PCA, practitioners often inspect the cumulative explained variance and keep enough components to reach roughly 90 to 99 percent of total variance, or look for the elbow in a scree plot. These heuristics are useful starting points, but the cross-validated score of the full pipeline is the more reliable criterion, because low-variance directions can still carry predictive signal.
The decision table below links typical situations to a sensible strategy.
| Situation | Recommended approach | Reason |
|---|---|---|
| Many correlated numeric features | PCA with whitening | Removes collinearity before linear models |
| Labeled data, few classes | LDA | Maximizes class separability directly |
| Exploratory cluster inspection | UMAP or t-SNE in 2D | Reveals local group structure visually |
| Sparse text or count matrices | Truncated SVD | Works without centering dense copies |
| Images or sensor streams | Autoencoder | Learns nonlinear features at scale |
| Very wide data, strict latency | Random projection | Distance-preserving and almost free |
Relation to feature selection and other neighboring concepts
Dimensionality reduction is an umbrella term, and feature selection and feature extraction are its two branches. Selection methods such as filter scores, wrapper searches or L1 regularization in lasso regression keep original variables, which matters when auditors or domain experts must read the model.
Reduction is also distinct from feature scaling, which changes the units of features but not their count. Scaling is nevertheless a prerequisite for variance-based methods: without standardization, PCA simply favors whichever feature has the largest numeric range. Clustering algorithms such as k-means often run on reduced data, because distances become more meaningful after noise dimensions are removed.
Limitations, risks and common mistakes
Every reduction discards information irreversibly, and the discarded part may be exactly what a later task needs. Nonlinear embeddings add further pitfalls, because their plots look persuasive even when the layout reflects hyperparameters more than data.
The most frequent errors and their remedies are listed below.
| Mistake | Consequence | Fix |
|---|---|---|
| Fitting the reducer on all data | Data leakage into validation scores | Fit inside each training fold only |
| Skipping standardization | Components dominated by large units | Scale features before PCA |
| Reading t-SNE cluster sizes | False conclusions about density | Treat only local proximity as evidence |
| Using t-SNE output as model input | No consistent transform for new data | Use PCA, UMAP or an encoder |
| Keeping components by variance alone | Loss of weak but predictive signal | Validate dimension on the task metric |
- Interpretability drops when extracted components mix dozens of original features.
- Nonlinear embeddings are stochastic, so results change with random seeds and perplexity or neighbor settings.
- Outliers distort linear projections, because variance is sensitive to extreme values; robust PCA variants mitigate this.
Understanding it simply
Imagine describing a house with two hundred measurements. Many of them repeat the same idea: floor area, number of rooms and total wall length all say how big the house is. Dimensionality reduction notices these overlaps and replaces them with a few summary numbers, such as size, age and location quality.
The computer then works with a handful of meaningful numbers instead of hundreds of noisy ones. It learns faster and makes fewer mistakes caused by clutter. The price is that some detail is always lost, so the summary must keep what matters for the question being asked.
Applied in practice
A data science team at a regional bank builds a credit risk model from 480 behavioral features describing card spending, balances and payment timing. Many features are strongly correlated, and the logistic regression model becomes unstable, with coefficients that flip sign between monthly retraining runs.
The team standardizes the features and fits PCA inside each cross-validation fold. Keeping 60 components retains about 95 percent of the variance, but validation shows that 40 components give the same area under the curve with lower variance across folds. For the internal review, the team also trains a lasso model on the original features to report which variables drive the scores.
The final pipeline trains in under a minute instead of twelve, its coefficients remain stable across retraining cycles, and a UMAP plot of the 40 components helps analysts spot a new cluster of thin-file applicants that needed a separate policy.