Principal component analysis

Also known as: PCA

On this page 7

Principal component analysis (PCA) is a linear technique that rotates a dataset into a new set of orthogonal axes ordered by explained variance, so that the first few axes, called principal components, capture most of the information in far fewer dimensions. In machine learning, PCA is the standard baseline for unsupervised dimensionality reduction, data compression, noise filtering and visualization.

PCA is unsupervised: it ignores class labels and looks only at the covariance structure of the features. That property makes it fast, deterministic and broadly applicable, but it also means the directions it keeps are not guaranteed to be the most useful ones for prediction.

How principal component analysis finds the components

PCA finds the directions along which the data varies most, subject to each new direction being perpendicular to all previous ones. Formally, the first principal component is the unit vector that maximizes the variance of the projected data; the second maximizes the remaining variance while staying orthogonal to the first, and so on.

The computation follows a fixed sequence. The data matrix is first centered by subtracting each feature mean, because PCA describes variation around the mean. The algorithm then computes either the eigendecomposition of the covariance matrix or, more commonly and more stably, the singular value decomposition (SVD) of the centered data matrix. The right singular vectors are the component directions, and the squared singular values divided by n minus 1 equal the eigenvalues, which measure the variance captured by each component.

The table below summarizes the main objects that appear in a PCA computation and what each one means.

QuantityMathematical meaningPractical interpretation
LoadingsEigenvectors of the covariance matrixWeights that combine original features into a component
EigenvaluesVariance along each componentHow much information each component carries
ScoresProjections of samples onto componentsNew coordinates of each data point
Explained variance ratioEigenvalue divided by total varianceShare of variation kept by one component
Reconstruction errorSquared distance to the projected pointInformation lost by truncation
Core quantities in a PCA computation

Keeping only the top k components gives the best rank-k linear approximation of the centered data in the least-squares sense, a result known as the Eckart-Young theorem. This is why PCA is simultaneously a variance-maximizing and an error-minimizing projection.

Choosing how many principal components to keep

The number of components k is the central hyperparameter of PCA, and no single rule is correct for every dataset. Practitioners usually combine a variance criterion with the needs of the downstream task.

Selection ruleHow it worksTypical caveat
Cumulative variance thresholdKeep components until 90 to 99 percent of variance is explainedThreshold choice is arbitrary
Scree plot elbowLook for the point where eigenvalues flattenElbow is often ambiguous
Kaiser criterionKeep eigenvalues above 1 on standardized dataTends to keep too many components
Cross-validationTune k by downstream model performanceCosts more computation
Parallel analysisCompare eigenvalues with those of random dataLess common in software defaults
Common rules for selecting the number of components

When PCA is a preprocessing step for a supervised model, treating k as a tuned hyperparameter inside a cross-validated pipeline is the most reliable approach. For visualization, two or three components are used regardless of how much variance they explain, and the explained share should be reported alongside the plot.

PCA is one of several methods for dimensionality reduction, and it differs from its neighbors in linearity, supervision and purpose. Singular value decomposition is the numerical engine behind PCA rather than a competing method, while the others answer different questions.

MethodLinearUses labelsMain purpose
PCAYesNoPreserve global variance
Linear discriminant analysisYesYesMaximize class separation
Kernel PCANoNoCapture nonlinear structure via a kernel
t-SNENoNoPreserve local neighborhoods for plots
UMAPNoNoFast neighborhood-preserving embedding
AutoencoderOptionalNoLearned nonlinear compression
PCA and neighboring techniques

A frequent pattern combines methods: PCA first reduces thousands of features to about 50 components to remove noise and speed up computation, and t-SNE or UMAP then produces the two-dimensional map. Unlike t-SNE, PCA preserves global distances and directions, so its axes remain interpretable and new samples can be projected with a single matrix multiplication.

Preprocessing and practical configuration

Correct preprocessing decides whether PCA results are meaningful. Because PCA is driven by variance, features measured in large units dominate the components unless the data is standardized to unit variance. Standardization is equivalent to running PCA on the correlation matrix instead of the covariance matrix.

  • Center every feature; most libraries do this automatically, but a manual SVD does not.
  • Standardize features with different units, such as income in dollars and age in years.
  • Keep raw scales when features share a unit and their magnitudes matter, as with pixel intensities or spectra.
  • Fit PCA on training data only and apply the same transformation to validation and test data to avoid leakage.
  • Use randomized SVD for very large matrices and incremental PCA when data does not fit in memory.

The computational cost of an exact SVD grows roughly as min(n p squared, n squared p) for n samples and p features, which is why randomized and truncated solvers are preferred when only a few components are needed from a wide matrix.

Limitations and common mistakes with PCA

PCA rests on assumptions that do not always hold. It captures only linear correlations, it equates high variance with importance, and it is sensitive to outliers, because a single extreme point can rotate a component toward itself. Robust PCA variants and outlier screening reduce this risk.

SymptomLikely causeFix
One feature dominates the first componentUnscaled features with large unitsStandardize before fitting
Optimistic test accuracyPCA fitted on the full datasetFit inside the training pipeline
Classes overlap after reductionDiscriminative signal has low varianceTry linear discriminant analysis or feature selection
Curved structure is flattenedNonlinear manifold in the dataUse kernel PCA or UMAP
Components change between runsSign ambiguity of eigenvectorsFix signs by convention before comparing
Typical PCA mistakes and how to fix them

Interpretability is another trade-off. Each component is a weighted mix of all original features, so explaining a model built on PCA scores is harder than explaining one built on selected features. Sparse PCA constrains loadings to few nonzero values when interpretability matters more than captured variance.

The short version

Imagine photographing a long, thin object such as a pencil. From most angles the picture shows very little, but from the side you see its full length. PCA looks for the most informative viewing angles of a dataset, the ones where the points are spread out the most.

It then keeps only the best few angles and drops the rest. A table with hundreds of columns can often be summarized by a handful of new columns that keep most of the information. The cost is that these new columns are blends of the old ones, so they are harder to name and explain.

Where it makes a difference

A manufacturing analytics team monitors a production line with 420 sensor readings per minute, covering temperatures, pressures and vibration levels. Many sensors move together, because one heater affects several thermometers at once. Training an anomaly detector on all 420 raw signals is slow and noisy.

The team standardizes the readings, fits PCA on three months of normal operation and finds that 18 components explain 95 percent of the variance. The detector then monitors two signals: the scores on those 18 components and the reconstruction error from the discarded ones. When a valve begins to stick, the reconstruction error rises sharply because the new pattern does not fit the normal correlations, and the alert arrives hours before any single sensor crosses its own limit. The outcome is faster training, fewer false alarms and earlier maintenance.