Singular value decomposition

Also known as: SVD

On this page 7

Singular value decomposition (SVD) is a matrix factorization that writes any real matrix A as the product U Σ Vᵀ, where U and V are orthogonal matrices and Σ is a diagonal matrix of non-negative singular values. In machine learning, SVD is the numerical workhorse behind principal component analysis, low-rank approximation, latent semantic analysis and many recommender system models.

Unlike eigendecomposition, SVD exists for every matrix of any shape, square or rectangular, full rank or rank-deficient. This universality, combined with excellent numerical stability, makes it the default tool whenever a data matrix must be compressed, denoised, inverted or analyzed for its dominant structure.

How singular value decomposition factors a matrix

SVD decomposes a matrix into three simple geometric operations: a rotation (Vᵀ), an axis-aligned scaling (Σ) and a second rotation (U). For an m × n matrix A, the columns of U are the left singular vectors, the columns of V are the right singular vectors, and the diagonal entries of Σ are the singular values, conventionally sorted from largest to smallest.

The factors are tied to two symmetric matrices. The right singular vectors are the eigenvectors of AᵀA, the left singular vectors are the eigenvectors of AAᵀ, and each singular value is the square root of an eigenvalue of AᵀA. The number of non-zero singular values equals the rank of the matrix, so SVD reveals rank directly.

The following table summarizes the components of the factorization and their roles.

ComponentShape (full SVD)Meaning
Um × mOrthonormal basis for the column space
Σm × nNon-negative singular values on the diagonal
Vᵀn × nOrthonormal basis for the row space
Rank rScalarCount of non-zero singular values
Components of the SVD of an m × n matrix

Full, thin and truncated SVD

Practical libraries offer several variants of SVD that differ in how many singular vectors they keep. The full SVD stores square U and V, which wastes memory for tall matrices; the thin SVD keeps only min(m, n) vectors; the truncated SVD keeps only the top k components and is the variant most used in machine learning.

The truncated form is justified by the Eckart-Young-Mirsky theorem, which states that keeping the k largest singular values gives the best rank-k approximation of A in both the Frobenius and spectral norms. No other rank-k matrix is closer to the original data.

VariantComponents keptTypical costTypical use
Full SVDAll m and n vectorsO(m n min(m, n))Theory, small matrices
Thin SVDmin(m, n) vectorsSame order, less memoryLeast squares, PCA on dense data
Truncated SVDTop k vectorsRoughly O(m n k)Compression, text, recommendations
Randomized SVDApproximate top kNear-linear in data sizeVery large or sparse matrices
SVD variants and their typical use

Randomized SVD projects the matrix onto a random low-dimensional subspace, orthogonalizes it and computes an exact SVD of the small result. With a few power iterations and oversampling, its accuracy is usually close to the exact truncated result at a fraction of the cost.

SVD compared with PCA and eigendecomposition

SVD is closely related to principal component analysis but is not the same concept. PCA is a statistical method that finds directions of maximum variance; SVD is the linear algebra routine that most PCA implementations use internally, applied to the mean-centered data matrix rather than to the covariance matrix.

Computing PCA through SVD avoids forming AᵀA explicitly, which would square the condition number and lose precision. The table contrasts the three related factorizations.

MethodApplies toOutputKey limitation
SVDAny m × n matrixTwo orthogonal bases and singular valuesCostly for very large dense data
EigendecompositionSquare, diagonalizable matricesEigenvectors and eigenvaluesMay be complex or not exist
PCAData sets of observationsVariance-ordered componentsRequires centering, assumes linearity
Non-negative matrix factorizationNon-negative matricesAdditive parts-based factorsNot unique, iterative and local
SVD versus related matrix methods

Other dimensionality reduction methods in the same family, such as t-SNE and UMAP, are nonlinear and stochastic. SVD remains linear, deterministic and exact, which makes it a reliable first step before trying more flexible methods.

Where SVD is used in machine learning

SVD appears wherever a model needs the dominant structure of a matrix. Its main applications include:

  • Dimensionality reduction: projecting features onto the top singular vectors before clustering or classification.
  • Latent semantic analysis: truncating a term-document matrix to uncover topics and synonym relationships.
  • Collaborative filtering: approximating a user-item rating matrix with latent factors.
  • Pseudoinverse and least squares: solving overdetermined linear systems and ridge regression stably.
  • Noise reduction: discarding small singular values that mostly carry noise.
  • Model compression: replacing large neural network weight matrices with low-rank factors.

Choosing the number of components k is the central practical decision. The table offers common strategies.

StrategyHow it worksWhen it fits
Explained energyKeep enough values to retain 90-99% of the squared sumGeneral compression and PCA
Scree or elbow inspectionCut where singular values flattenExploratory analysis
Cross-validationPick k that maximizes downstream accuracySupervised pipelines
Noise thresholdDrop values below an estimated noise floorDenoising signals and images
Strategies for choosing the truncation rank

Limitations and common mistakes

SVD is powerful but sensitive to preprocessing and scale. Its main pitfalls are well known to practitioners:

  • Forgetting to center the data turns the first component into a mean direction instead of a variance direction.
  • Mixing feature scales lets large-unit variables dominate; standardize when units differ.
  • Treating singular vectors as unique ignores sign flips and rotations within repeated singular values.
  • Imputing missing values with zeros biases recommender factors; use masked matrix factorization instead.
  • Running full SVD on huge matrices wastes memory; use truncated or randomized solvers.

SVD also captures only linear structure and is not robust to outliers, because a single extreme value can rotate the leading singular vectors. Robust PCA and kernel methods address these cases at higher computational cost.

In simple terms

Singular value decomposition takes a big table of numbers and splits it into a few simple building blocks. Each block describes one pattern in the data, and each comes with a number that says how important that pattern is.

Think of a music mix. SVD separates the mix into individual tracks and tells you how loud each one is. If you keep only the loudest tracks, you still hear the song almost perfectly, but you store far less. That is why SVD is used to shrink data while keeping what matters.

A practical example

A data science team at an online bookstore holds a rating matrix of 200,000 customers by 50,000 titles, in which more than 99% of the cells are empty. They want recommendations that go beyond simple bestseller lists.

The team fits a truncated, SVD-style factorization with 64 latent factors, training only on the observed ratings and adding regularization to avoid overfitting. Each customer and each book becomes a short vector of 64 numbers, and the predicted rating is the dot product of the two vectors. Inspection shows that early factors separate genres such as fantasy and business writing.

On a held-out set of ratings, the model lowers prediction error noticeably compared with a baseline based on item averages. The result is personalized recommendations computed in milliseconds, with storage reduced from billions of potential cells to a few million factor values.