Linear discriminant analysis (LDA) is a supervised method that projects data onto directions which maximize separation between known classes, and it serves both as a classifier and as a dimensionality reduction technique in machine learning. The two-class version is also called Fisher’s linear discriminant, after the statistician who introduced it for separating plant species from measurements.
Within artificial intelligence, LDA belongs to the family of generative classifiers: it models how each class produces its features and then applies Bayes’ rule to assign new observations. Its core assumption is that every class follows a multivariate normal distribution with a shared covariance matrix, which is exactly what makes the resulting decision boundaries straight lines or flat hyperplanes.
How linear discriminant analysis separates classes
LDA separates classes by finding projection directions along which the class means lie far apart while each class stays compact. Fisher expressed this as a ratio: the between-class scatter divided by the within-class scatter. The direction that maximizes this ratio is the most discriminating view of the data.
Mathematically, the solution comes from a generalized eigenvalue problem involving the within-class scatter matrix and the between-class scatter matrix. The eigenvectors with the largest eigenvalues become the discriminant axes, and their eigenvalues measure how much separating power each axis carries. For two classes, the single optimal direction is simply the inverse of the pooled covariance multiplied by the difference of the two class means.
The following table lists the main quantities LDA estimates from training data and what each one controls.
| Quantity | What it describes | Role in the model |
|---|---|---|
| Class means | Average feature vector of each class | Locate class centers in feature space |
| Pooled covariance | Shared spread and correlation of features | Defines distance and shape of class clouds |
| Class priors | Share of each class in the population | Shift the boundary toward rarer classes |
| Discriminant vectors | Directions of maximal class separation | Used for projection and scoring |
| Eigenvalues | Separating power of each direction | Rank axes and guide how many to keep |
Classification and dimensionality reduction with LDA
LDA can be used in two distinct roles, and practitioners often combine them. As a classifier, it computes a linear discriminant score for each class and assigns the observation to the class with the highest posterior probability. As a projection method, it maps data onto at most K minus 1 dimensions, where K is the number of classes, because the between-class scatter matrix has at most that rank.
This rank limit is a frequent surprise: a problem with three classes yields at most two discriminant axes, regardless of whether the data have ten or ten thousand features. That makes LDA an aggressive but highly targeted form of dimensionality reduction, well suited to visualization and to compact inputs for downstream models.
- Classifier output: a predicted class and calibrated posterior probabilities when the assumptions hold.
- Projection output: low-dimensional coordinates that preserve class structure for plotting or further modeling.
- Shrinkage variant: regularizes the covariance estimate so the method remains stable when features outnumber samples.
How LDA compares with related methods
LDA is best understood against its neighbors in the machine learning toolbox, because each makes different assumptions about the data. The table below contrasts it with the methods most often considered for the same task.
| Method | Uses labels | Boundary shape | Key assumption or goal |
|---|---|---|---|
| Linear discriminant analysis | Yes | Linear | Gaussian classes with shared covariance |
| Quadratic discriminant analysis | Yes | Quadratic | Gaussian classes with separate covariances |
| Principal component analysis | No | Not a classifier | Maximize total variance |
| Logistic regression | Yes | Linear | Models class probability directly |
| Naive Bayes classifier | Yes | Often nonlinear | Features independent within each class |
The contrast with principal component analysis matters most. PCA ignores labels and keeps directions of maximal variance, which may carry no class information at all; LDA keeps the directions that separate labeled groups. Compared with logistic regression, LDA is more statistically efficient when its Gaussian assumptions hold, but logistic regression is more robust when they do not, since it never models the feature distribution.
When to choose linear discriminant analysis
LDA is a strong choice when classes are roughly elliptical, similarly shaped and linearly separable, and when training data are limited. Because it estimates only means and one shared covariance, it needs far fewer parameters than flexible models and trains in a single closed-form step without iterative optimization.
The decision table below summarizes common situations and the recommended approach.
| Situation | Recommended choice | Reason |
|---|---|---|
| Small sample, near-Gaussian features | Standard LDA | Few parameters, stable estimates |
| More features than samples | Shrinkage LDA | Regularized covariance stays invertible |
| Classes with very different spreads | Quadratic discriminant analysis | Allows class-specific covariance |
| Heavy outliers or skewed features | Logistic regression | Does not assume normal features |
| Complex nonlinear boundaries | Kernel methods or tree ensembles | Capture curved class regions |
Before fitting, standardize features if they are used for interpretation, inspect class covariance matrices for gross differences, and check that no class is tiny, since covariance estimates degrade quickly with few samples.
Limitations and common mistakes
The main weakness of LDA is its reliance on the shared covariance assumption. When one class is tightly clustered and another widely spread, the linear boundary sits in the wrong place and accuracy drops. LDA is also sensitive to outliers, because means and covariances are not robust statistics.
| Symptom | Likely cause | Fix |
|---|---|---|
| Singular matrix error | More features than samples | Apply shrinkage or reduce features first |
| Good training, poor test accuracy | Unstable covariance estimate | Regularize and cross-validate |
| Minority class rarely predicted | Skewed class priors | Adjust priors or decision threshold |
| Boundary misses curved structure | Nonlinear class regions | Switch to QDA or a kernel model |
| Projection shows no separation | Labels unrelated to features | Revisit feature engineering |
A common misconception is that LDA requires perfectly normal data. In practice it is reasonably robust to moderate departures from normality, but it is not robust to strongly unequal covariances, heavy tails or categorical features coded as numbers. Another error is confusing it with latent Dirichlet allocation, an unrelated topic model that shares the same acronym.
In simple terms
Imagine sorting apples and pears by weight, color and shape. Linear discriminant analysis looks for the single best angle to view the fruit. From that angle, apples bunch together on one side. Pears gather on the other side, with little overlap.
Sorting a new piece of fruit is then easy. Place it on the same scale and see which group is closer. The method works best when each group forms a tidy cloud. It struggles when groups are oddly shaped or mixed together.
A practical example
A quality engineer at a food processing plant needs to classify olive oil samples into three production regions using eight chemical measurements, such as fatty acid percentages. Only about 180 labeled samples exist, too few to train a large model reliably.
The engineer standardizes the measurements and fits shrinkage LDA. Because there are three regions, the method produces two discriminant axes, and a scatter plot of those axes shows three clearly separated clusters, which helps auditors understand the result. Cross-validated accuracy reaches about 95 percent, and the eigenvalues show that the first axis carries most of the separating power.
The outcome is a fast, transparent screening tool: new samples are classified in milliseconds, suspicious samples falling between clusters are flagged for laboratory review, and the linear coefficients reveal which chemical markers distinguish each region.