Linear discriminant analysis

Also known as: Fisher's linear discriminant

On this page 7

Linear discriminant analysis (LDA) is a supervised method that projects data onto directions which maximize separation between known classes, and it serves both as a classifier and as a dimensionality reduction technique in machine learning. The two-class version is also called Fisher’s linear discriminant, after the statistician who introduced it for separating plant species from measurements.

Within artificial intelligence, LDA belongs to the family of generative classifiers: it models how each class produces its features and then applies Bayes’ rule to assign new observations. Its core assumption is that every class follows a multivariate normal distribution with a shared covariance matrix, which is exactly what makes the resulting decision boundaries straight lines or flat hyperplanes.

How linear discriminant analysis separates classes

LDA separates classes by finding projection directions along which the class means lie far apart while each class stays compact. Fisher expressed this as a ratio: the between-class scatter divided by the within-class scatter. The direction that maximizes this ratio is the most discriminating view of the data.

Mathematically, the solution comes from a generalized eigenvalue problem involving the within-class scatter matrix and the between-class scatter matrix. The eigenvectors with the largest eigenvalues become the discriminant axes, and their eigenvalues measure how much separating power each axis carries. For two classes, the single optimal direction is simply the inverse of the pooled covariance multiplied by the difference of the two class means.

The following table lists the main quantities LDA estimates from training data and what each one controls.

QuantityWhat it describesRole in the model
Class meansAverage feature vector of each classLocate class centers in feature space
Pooled covarianceShared spread and correlation of featuresDefines distance and shape of class clouds
Class priorsShare of each class in the populationShift the boundary toward rarer classes
Discriminant vectorsDirections of maximal class separationUsed for projection and scoring
EigenvaluesSeparating power of each directionRank axes and guide how many to keep
Quantities estimated by linear discriminant analysis

Classification and dimensionality reduction with LDA

LDA can be used in two distinct roles, and practitioners often combine them. As a classifier, it computes a linear discriminant score for each class and assigns the observation to the class with the highest posterior probability. As a projection method, it maps data onto at most K minus 1 dimensions, where K is the number of classes, because the between-class scatter matrix has at most that rank.

This rank limit is a frequent surprise: a problem with three classes yields at most two discriminant axes, regardless of whether the data have ten or ten thousand features. That makes LDA an aggressive but highly targeted form of dimensionality reduction, well suited to visualization and to compact inputs for downstream models.

  • Classifier output: a predicted class and calibrated posterior probabilities when the assumptions hold.
  • Projection output: low-dimensional coordinates that preserve class structure for plotting or further modeling.
  • Shrinkage variant: regularizes the covariance estimate so the method remains stable when features outnumber samples.

LDA is best understood against its neighbors in the machine learning toolbox, because each makes different assumptions about the data. The table below contrasts it with the methods most often considered for the same task.

MethodUses labelsBoundary shapeKey assumption or goal
Linear discriminant analysisYesLinearGaussian classes with shared covariance
Quadratic discriminant analysisYesQuadraticGaussian classes with separate covariances
Principal component analysisNoNot a classifierMaximize total variance
Logistic regressionYesLinearModels class probability directly
Naive Bayes classifierYesOften nonlinearFeatures independent within each class
LDA compared with related techniques

The contrast with principal component analysis matters most. PCA ignores labels and keeps directions of maximal variance, which may carry no class information at all; LDA keeps the directions that separate labeled groups. Compared with logistic regression, LDA is more statistically efficient when its Gaussian assumptions hold, but logistic regression is more robust when they do not, since it never models the feature distribution.

When to choose linear discriminant analysis

LDA is a strong choice when classes are roughly elliptical, similarly shaped and linearly separable, and when training data are limited. Because it estimates only means and one shared covariance, it needs far fewer parameters than flexible models and trains in a single closed-form step without iterative optimization.

The decision table below summarizes common situations and the recommended approach.

SituationRecommended choiceReason
Small sample, near-Gaussian featuresStandard LDAFew parameters, stable estimates
More features than samplesShrinkage LDARegularized covariance stays invertible
Classes with very different spreadsQuadratic discriminant analysisAllows class-specific covariance
Heavy outliers or skewed featuresLogistic regressionDoes not assume normal features
Complex nonlinear boundariesKernel methods or tree ensemblesCapture curved class regions
Choosing between LDA and its alternatives

Before fitting, standardize features if they are used for interpretation, inspect class covariance matrices for gross differences, and check that no class is tiny, since covariance estimates degrade quickly with few samples.

Limitations and common mistakes

The main weakness of LDA is its reliance on the shared covariance assumption. When one class is tightly clustered and another widely spread, the linear boundary sits in the wrong place and accuracy drops. LDA is also sensitive to outliers, because means and covariances are not robust statistics.

SymptomLikely causeFix
Singular matrix errorMore features than samplesApply shrinkage or reduce features first
Good training, poor test accuracyUnstable covariance estimateRegularize and cross-validate
Minority class rarely predictedSkewed class priorsAdjust priors or decision threshold
Boundary misses curved structureNonlinear class regionsSwitch to QDA or a kernel model
Projection shows no separationLabels unrelated to featuresRevisit feature engineering
Typical problems with LDA and how to address them

A common misconception is that LDA requires perfectly normal data. In practice it is reasonably robust to moderate departures from normality, but it is not robust to strongly unequal covariances, heavy tails or categorical features coded as numbers. Another error is confusing it with latent Dirichlet allocation, an unrelated topic model that shares the same acronym.

In simple terms

Imagine sorting apples and pears by weight, color and shape. Linear discriminant analysis looks for the single best angle to view the fruit. From that angle, apples bunch together on one side. Pears gather on the other side, with little overlap.

Sorting a new piece of fruit is then easy. Place it on the same scale and see which group is closer. The method works best when each group forms a tidy cloud. It struggles when groups are oddly shaped or mixed together.

A practical example

A quality engineer at a food processing plant needs to classify olive oil samples into three production regions using eight chemical measurements, such as fatty acid percentages. Only about 180 labeled samples exist, too few to train a large model reliably.

The engineer standardizes the measurements and fits shrinkage LDA. Because there are three regions, the method produces two discriminant axes, and a scatter plot of those axes shows three clearly separated clusters, which helps auditors understand the result. Cross-validated accuracy reaches about 95 percent, and the eigenvalues show that the first axis carries most of the separating power.

The outcome is a fast, transparent screening tool: new samples are classified in milliseconds, suspicious samples falling between clusters are flagged for laboratory review, and the linear coefficients reveal which chemical markers distinguish each region.