Singular value decomposition (SVD) is a matrix factorization that writes any real matrix A as the product U Σ Vᵀ, where U and V are orthogonal matrices and Σ is a diagonal matrix of non-negative singular values. In machine learning, SVD is the numerical workhorse behind principal component analysis, low-rank approximation, latent semantic analysis and many recommender system models.
Unlike eigendecomposition, SVD exists for every matrix of any shape, square or rectangular, full rank or rank-deficient. This universality, combined with excellent numerical stability, makes it the default tool whenever a data matrix must be compressed, denoised, inverted or analyzed for its dominant structure.
How singular value decomposition factors a matrix
SVD decomposes a matrix into three simple geometric operations: a rotation (Vᵀ), an axis-aligned scaling (Σ) and a second rotation (U). For an m × n matrix A, the columns of U are the left singular vectors, the columns of V are the right singular vectors, and the diagonal entries of Σ are the singular values, conventionally sorted from largest to smallest.
The factors are tied to two symmetric matrices. The right singular vectors are the eigenvectors of AᵀA, the left singular vectors are the eigenvectors of AAᵀ, and each singular value is the square root of an eigenvalue of AᵀA. The number of non-zero singular values equals the rank of the matrix, so SVD reveals rank directly.
The following table summarizes the components of the factorization and their roles.
| Component | Shape (full SVD) | Meaning |
|---|---|---|
| U | m × m | Orthonormal basis for the column space |
| Σ | m × n | Non-negative singular values on the diagonal |
| Vᵀ | n × n | Orthonormal basis for the row space |
| Rank r | Scalar | Count of non-zero singular values |
Full, thin and truncated SVD
Practical libraries offer several variants of SVD that differ in how many singular vectors they keep. The full SVD stores square U and V, which wastes memory for tall matrices; the thin SVD keeps only min(m, n) vectors; the truncated SVD keeps only the top k components and is the variant most used in machine learning.
The truncated form is justified by the Eckart-Young-Mirsky theorem, which states that keeping the k largest singular values gives the best rank-k approximation of A in both the Frobenius and spectral norms. No other rank-k matrix is closer to the original data.
| Variant | Components kept | Typical cost | Typical use |
|---|---|---|---|
| Full SVD | All m and n vectors | O(m n min(m, n)) | Theory, small matrices |
| Thin SVD | min(m, n) vectors | Same order, less memory | Least squares, PCA on dense data |
| Truncated SVD | Top k vectors | Roughly O(m n k) | Compression, text, recommendations |
| Randomized SVD | Approximate top k | Near-linear in data size | Very large or sparse matrices |
Randomized SVD projects the matrix onto a random low-dimensional subspace, orthogonalizes it and computes an exact SVD of the small result. With a few power iterations and oversampling, its accuracy is usually close to the exact truncated result at a fraction of the cost.
SVD compared with PCA and eigendecomposition
SVD is closely related to principal component analysis but is not the same concept. PCA is a statistical method that finds directions of maximum variance; SVD is the linear algebra routine that most PCA implementations use internally, applied to the mean-centered data matrix rather than to the covariance matrix.
Computing PCA through SVD avoids forming AᵀA explicitly, which would square the condition number and lose precision. The table contrasts the three related factorizations.
| Method | Applies to | Output | Key limitation |
|---|---|---|---|
| SVD | Any m × n matrix | Two orthogonal bases and singular values | Costly for very large dense data |
| Eigendecomposition | Square, diagonalizable matrices | Eigenvectors and eigenvalues | May be complex or not exist |
| PCA | Data sets of observations | Variance-ordered components | Requires centering, assumes linearity |
| Non-negative matrix factorization | Non-negative matrices | Additive parts-based factors | Not unique, iterative and local |
Other dimensionality reduction methods in the same family, such as t-SNE and UMAP, are nonlinear and stochastic. SVD remains linear, deterministic and exact, which makes it a reliable first step before trying more flexible methods.
Where SVD is used in machine learning
SVD appears wherever a model needs the dominant structure of a matrix. Its main applications include:
- Dimensionality reduction: projecting features onto the top singular vectors before clustering or classification.
- Latent semantic analysis: truncating a term-document matrix to uncover topics and synonym relationships.
- Collaborative filtering: approximating a user-item rating matrix with latent factors.
- Pseudoinverse and least squares: solving overdetermined linear systems and ridge regression stably.
- Noise reduction: discarding small singular values that mostly carry noise.
- Model compression: replacing large neural network weight matrices with low-rank factors.
Choosing the number of components k is the central practical decision. The table offers common strategies.
| Strategy | How it works | When it fits |
|---|---|---|
| Explained energy | Keep enough values to retain 90-99% of the squared sum | General compression and PCA |
| Scree or elbow inspection | Cut where singular values flatten | Exploratory analysis |
| Cross-validation | Pick k that maximizes downstream accuracy | Supervised pipelines |
| Noise threshold | Drop values below an estimated noise floor | Denoising signals and images |
Limitations and common mistakes
SVD is powerful but sensitive to preprocessing and scale. Its main pitfalls are well known to practitioners:
- Forgetting to center the data turns the first component into a mean direction instead of a variance direction.
- Mixing feature scales lets large-unit variables dominate; standardize when units differ.
- Treating singular vectors as unique ignores sign flips and rotations within repeated singular values.
- Imputing missing values with zeros biases recommender factors; use masked matrix factorization instead.
- Running full SVD on huge matrices wastes memory; use truncated or randomized solvers.
SVD also captures only linear structure and is not robust to outliers, because a single extreme value can rotate the leading singular vectors. Robust PCA and kernel methods address these cases at higher computational cost.
In simple terms
Singular value decomposition takes a big table of numbers and splits it into a few simple building blocks. Each block describes one pattern in the data, and each comes with a number that says how important that pattern is.
Think of a music mix. SVD separates the mix into individual tracks and tells you how loud each one is. If you keep only the loudest tracks, you still hear the song almost perfectly, but you store far less. That is why SVD is used to shrink data while keeping what matters.
A practical example
A data science team at an online bookstore holds a rating matrix of 200,000 customers by 50,000 titles, in which more than 99% of the cells are empty. They want recommendations that go beyond simple bestseller lists.
The team fits a truncated, SVD-style factorization with 64 latent factors, training only on the observed ratings and adding regularization to avoid overfitting. Each customer and each book becomes a short vector of 64 numbers, and the predicted rating is the dot product of the two vectors. Inspection shows that early factors separate genres such as fantasy and business writing.
On a held-out set of ratings, the model lowers prediction error noticeably compared with a baseline based on item averages. The result is personalized recommendations computed in milliseconds, with storage reduced from billions of potential cells to a few million factor values.