Principal component analysis (PCA) is a linear technique that rotates a dataset into a new set of orthogonal axes ordered by explained variance, so that the first few axes, called principal components, capture most of the information in far fewer dimensions. In machine learning, PCA is the standard baseline for unsupervised dimensionality reduction, data compression, noise filtering and visualization.
PCA is unsupervised: it ignores class labels and looks only at the covariance structure of the features. That property makes it fast, deterministic and broadly applicable, but it also means the directions it keeps are not guaranteed to be the most useful ones for prediction.
How principal component analysis finds the components
PCA finds the directions along which the data varies most, subject to each new direction being perpendicular to all previous ones. Formally, the first principal component is the unit vector that maximizes the variance of the projected data; the second maximizes the remaining variance while staying orthogonal to the first, and so on.
The computation follows a fixed sequence. The data matrix is first centered by subtracting each feature mean, because PCA describes variation around the mean. The algorithm then computes either the eigendecomposition of the covariance matrix or, more commonly and more stably, the singular value decomposition (SVD) of the centered data matrix. The right singular vectors are the component directions, and the squared singular values divided by n minus 1 equal the eigenvalues, which measure the variance captured by each component.
The table below summarizes the main objects that appear in a PCA computation and what each one means.
| Quantity | Mathematical meaning | Practical interpretation |
|---|---|---|
| Loadings | Eigenvectors of the covariance matrix | Weights that combine original features into a component |
| Eigenvalues | Variance along each component | How much information each component carries |
| Scores | Projections of samples onto components | New coordinates of each data point |
| Explained variance ratio | Eigenvalue divided by total variance | Share of variation kept by one component |
| Reconstruction error | Squared distance to the projected point | Information lost by truncation |
Keeping only the top k components gives the best rank-k linear approximation of the centered data in the least-squares sense, a result known as the Eckart-Young theorem. This is why PCA is simultaneously a variance-maximizing and an error-minimizing projection.
Choosing how many principal components to keep
The number of components k is the central hyperparameter of PCA, and no single rule is correct for every dataset. Practitioners usually combine a variance criterion with the needs of the downstream task.
| Selection rule | How it works | Typical caveat |
|---|---|---|
| Cumulative variance threshold | Keep components until 90 to 99 percent of variance is explained | Threshold choice is arbitrary |
| Scree plot elbow | Look for the point where eigenvalues flatten | Elbow is often ambiguous |
| Kaiser criterion | Keep eigenvalues above 1 on standardized data | Tends to keep too many components |
| Cross-validation | Tune k by downstream model performance | Costs more computation |
| Parallel analysis | Compare eigenvalues with those of random data | Less common in software defaults |
When PCA is a preprocessing step for a supervised model, treating k as a tuned hyperparameter inside a cross-validated pipeline is the most reliable approach. For visualization, two or three components are used regardless of how much variance they explain, and the explained share should be reported alongside the plot.
PCA compared with related dimensionality reduction methods
PCA is one of several methods for dimensionality reduction, and it differs from its neighbors in linearity, supervision and purpose. Singular value decomposition is the numerical engine behind PCA rather than a competing method, while the others answer different questions.
| Method | Linear | Uses labels | Main purpose |
|---|---|---|---|
| PCA | Yes | No | Preserve global variance |
| Linear discriminant analysis | Yes | Yes | Maximize class separation |
| Kernel PCA | No | No | Capture nonlinear structure via a kernel |
| t-SNE | No | No | Preserve local neighborhoods for plots |
| UMAP | No | No | Fast neighborhood-preserving embedding |
| Autoencoder | Optional | No | Learned nonlinear compression |
A frequent pattern combines methods: PCA first reduces thousands of features to about 50 components to remove noise and speed up computation, and t-SNE or UMAP then produces the two-dimensional map. Unlike t-SNE, PCA preserves global distances and directions, so its axes remain interpretable and new samples can be projected with a single matrix multiplication.
Preprocessing and practical configuration
Correct preprocessing decides whether PCA results are meaningful. Because PCA is driven by variance, features measured in large units dominate the components unless the data is standardized to unit variance. Standardization is equivalent to running PCA on the correlation matrix instead of the covariance matrix.
- Center every feature; most libraries do this automatically, but a manual SVD does not.
- Standardize features with different units, such as income in dollars and age in years.
- Keep raw scales when features share a unit and their magnitudes matter, as with pixel intensities or spectra.
- Fit PCA on training data only and apply the same transformation to validation and test data to avoid leakage.
- Use randomized SVD for very large matrices and incremental PCA when data does not fit in memory.
The computational cost of an exact SVD grows roughly as min(n p squared, n squared p) for n samples and p features, which is why randomized and truncated solvers are preferred when only a few components are needed from a wide matrix.
Limitations and common mistakes with PCA
PCA rests on assumptions that do not always hold. It captures only linear correlations, it equates high variance with importance, and it is sensitive to outliers, because a single extreme point can rotate a component toward itself. Robust PCA variants and outlier screening reduce this risk.
| Symptom | Likely cause | Fix |
|---|---|---|
| One feature dominates the first component | Unscaled features with large units | Standardize before fitting |
| Optimistic test accuracy | PCA fitted on the full dataset | Fit inside the training pipeline |
| Classes overlap after reduction | Discriminative signal has low variance | Try linear discriminant analysis or feature selection |
| Curved structure is flattened | Nonlinear manifold in the data | Use kernel PCA or UMAP |
| Components change between runs | Sign ambiguity of eigenvectors | Fix signs by convention before comparing |
Interpretability is another trade-off. Each component is a weighted mix of all original features, so explaining a model built on PCA scores is harder than explaining one built on selected features. Sparse PCA constrains loadings to few nonzero values when interpretability matters more than captured variance.
The short version
Imagine photographing a long, thin object such as a pencil. From most angles the picture shows very little, but from the side you see its full length. PCA looks for the most informative viewing angles of a dataset, the ones where the points are spread out the most.
It then keeps only the best few angles and drops the rest. A table with hundreds of columns can often be summarized by a handful of new columns that keep most of the information. The cost is that these new columns are blends of the old ones, so they are harder to name and explain.
Where it makes a difference
A manufacturing analytics team monitors a production line with 420 sensor readings per minute, covering temperatures, pressures and vibration levels. Many sensors move together, because one heater affects several thermometers at once. Training an anomaly detector on all 420 raw signals is slow and noisy.
The team standardizes the readings, fits PCA on three months of normal operation and finds that 18 components explain 95 percent of the variance. The detector then monitors two signals: the scores on those 18 components and the reconstruction error from the discarded ones. When a valve begins to stick, the reconstruction error rises sharply because the new pattern does not fit the normal correlations, and the alert arrives hours before any single sensor crosses its own limit. The outcome is faster training, fewer false alarms and earlier maintenance.