Dimensionality reduction

On this page 7

Dimensionality reduction is the family of machine learning techniques that transform data with many features into a representation with far fewer dimensions while preserving the structure that matters for a given task, such as variance, distances, class separation or neighborhood relations. In artificial intelligence practice, it serves three distinct purposes: compressing inputs for faster and more stable models, visualizing high-dimensional data in two or three dimensions, and removing noise and redundancy before clustering or classification.

The method answers a practical problem known as the curse of dimensionality: as the number of features grows, data points become sparse, distances lose contrast and models need exponentially more samples to generalize. Reducing dimensions trades some information for better statistical efficiency, lower computational cost and clearer interpretation.

How dimensionality reduction works

Every dimensionality reduction method learns a mapping from the original feature space to a lower-dimensional space, guided by an objective that defines what must survive the compression. Real data rarely fills its full feature space; it tends to lie near a low-dimensional manifold, and the methods try to recover the coordinates of that manifold.

There are two broad strategies. Feature selection keeps a subset of the original columns and discards the rest, so the retained features stay interpretable. Feature extraction builds new variables as combinations or nonlinear functions of all inputs, which usually preserves more information but produces components without a direct physical meaning.

The table below summarizes the main objectives that different methods optimize.

ObjectiveWhat is preservedTypical method
Maximum varianceDirections with the largest spreadPrincipal component analysis
Class separationRatio of between-class to within-class scatterLinear discriminant analysis
Local neighborhoodsWhich points are near each othert-SNE, UMAP
Reconstruction accuracyAbility to rebuild the inputAutoencoder
Pairwise distancesGlobal distance structureMultidimensional scaling, random projection
Objectives behind common reduction methods

Linear and nonlinear methods

Linear methods project data onto a hyperplane spanned by a few basis vectors, which makes them fast, deterministic and reversible up to the discarded components. Principal component analysis (PCA) is the reference case: it computes the eigenvectors of the covariance matrix, usually through singular value decomposition, and orders them by the variance they explain.

Nonlinear methods handle data that lies on curved manifolds, where a straight projection would fold distant points on top of each other. Kernel PCA applies the kernel trick, Isomap preserves geodesic distances along a neighborhood graph, and t-SNE and UMAP optimize low-dimensional embeddings so that local neighbors stay close. Neural autoencoders learn an encoder and decoder jointly and scale to images, audio and text.

The following comparison helps choose among the most frequently used techniques.

MethodTypeSupervisionNew dataMain strength
PCALinearUnsupervisedDirect projectionFast, interpretable variance ranking
LDALinearSupervisedDirect projectionAt most C minus 1 discriminative axes
Kernel PCANonlinearUnsupervisedKernel evaluationCaptures curved structure
t-SNENonlinearUnsupervisedNo native transformSharp local cluster views
UMAPNonlinearOptionalApproximate transformFaster, better global layout
AutoencoderNonlinearSelf-supervisedEncoder passScales to raw signals
Comparison of widely used techniques

Choosing the target number of dimensions

The number of output dimensions is a hyperparameter that depends on the downstream goal, not a property of the data alone. For visualization, the answer is fixed at two or three dimensions. For modeling, the dimension should be selected by validation performance or by an explicit information budget.

With PCA, practitioners often inspect the cumulative explained variance and keep enough components to reach roughly 90 to 99 percent of total variance, or look for the elbow in a scree plot. These heuristics are useful starting points, but the cross-validated score of the full pipeline is the more reliable criterion, because low-variance directions can still carry predictive signal.

The decision table below links typical situations to a sensible strategy.

SituationRecommended approachReason
Many correlated numeric featuresPCA with whiteningRemoves collinearity before linear models
Labeled data, few classesLDAMaximizes class separability directly
Exploratory cluster inspectionUMAP or t-SNE in 2DReveals local group structure visually
Sparse text or count matricesTruncated SVDWorks without centering dense copies
Images or sensor streamsAutoencoderLearns nonlinear features at scale
Very wide data, strict latencyRandom projectionDistance-preserving and almost free
Decision guide for practical projects

Relation to feature selection and other neighboring concepts

Dimensionality reduction is an umbrella term, and feature selection and feature extraction are its two branches. Selection methods such as filter scores, wrapper searches or L1 regularization in lasso regression keep original variables, which matters when auditors or domain experts must read the model.

Reduction is also distinct from feature scaling, which changes the units of features but not their count. Scaling is nevertheless a prerequisite for variance-based methods: without standardization, PCA simply favors whichever feature has the largest numeric range. Clustering algorithms such as k-means often run on reduced data, because distances become more meaningful after noise dimensions are removed.

Limitations, risks and common mistakes

Every reduction discards information irreversibly, and the discarded part may be exactly what a later task needs. Nonlinear embeddings add further pitfalls, because their plots look persuasive even when the layout reflects hyperparameters more than data.

The most frequent errors and their remedies are listed below.

MistakeConsequenceFix
Fitting the reducer on all dataData leakage into validation scoresFit inside each training fold only
Skipping standardizationComponents dominated by large unitsScale features before PCA
Reading t-SNE cluster sizesFalse conclusions about densityTreat only local proximity as evidence
Using t-SNE output as model inputNo consistent transform for new dataUse PCA, UMAP or an encoder
Keeping components by variance aloneLoss of weak but predictive signalValidate dimension on the task metric
Typical errors and how to correct them
  • Interpretability drops when extracted components mix dozens of original features.
  • Nonlinear embeddings are stochastic, so results change with random seeds and perplexity or neighbor settings.
  • Outliers distort linear projections, because variance is sensitive to extreme values; robust PCA variants mitigate this.

Understanding it simply

Imagine describing a house with two hundred measurements. Many of them repeat the same idea: floor area, number of rooms and total wall length all say how big the house is. Dimensionality reduction notices these overlaps and replaces them with a few summary numbers, such as size, age and location quality.

The computer then works with a handful of meaningful numbers instead of hundreds of noisy ones. It learns faster and makes fewer mistakes caused by clutter. The price is that some detail is always lost, so the summary must keep what matters for the question being asked.

Applied in practice

A data science team at a regional bank builds a credit risk model from 480 behavioral features describing card spending, balances and payment timing. Many features are strongly correlated, and the logistic regression model becomes unstable, with coefficients that flip sign between monthly retraining runs.

The team standardizes the features and fits PCA inside each cross-validation fold. Keeping 60 components retains about 95 percent of the variance, but validation shows that 40 components give the same area under the curve with lower variance across folds. For the internal review, the team also trains a lasso model on the original features to report which variables drive the scores.

The final pipeline trains in under a minute instead of twelve, its coefficients remain stable across retraining cycles, and a UMAP plot of the 40 components helps analysts spot a new cluster of thin-file applicants that needed a separate policy.