Ridge regression

Also known as: Tikhonov regularization

On this page 7

Ridge regression is a form of linear regression that adds a penalty on the squared size of the coefficients to the ordinary least squares loss, shrinking all weights toward zero to reduce variance and stabilize estimates. Machine learning practitioners also know it as Tikhonov regularization, the name used in numerical analysis and inverse problems, where the same idea was developed to solve ill-conditioned systems.

Within classic machine learning, ridge is the default answer when a linear model has many correlated features or more features than reliable observations. It belongs to the family of regularization methods, alongside lasso regression and elastic net, and it is one of the clearest practical illustrations of the bias-variance tradeoff: a small, deliberate bias buys a large reduction in prediction error on new data.

How the L2 penalty reshapes the least squares problem

Ridge regression minimizes the residual sum of squares plus lambda times the sum of squared coefficients. Written compactly, the objective is the squared error term plus lambda * ||w||^2, where lambda (also called alpha) is a non-negative hyperparameter that sets the strength of the penalty. The intercept is normally left unpenalized, because shrinking it would only shift predictions without controlling complexity.

The penalty has a closed-form solution: w = (X^T X + lambda I)^-1 X^T y. Adding lambda to the diagonal of X^T X makes the matrix strictly positive definite and always invertible, even when features are perfectly collinear or when the number of features exceeds the number of samples. This is exactly the numerical repair that Tikhonov introduced for ill-posed problems.

Viewed through the singular value decomposition of the design matrix, ridge multiplies each component by a shrinkage factor of d squared over (d squared plus lambda), where d is the singular value. Directions with large variance in the data are barely touched, while low-variance, noisy directions are shrunk heavily. The table below summarizes how the main quantities behave as lambda changes.

Lambda settingCoefficientsBiasVarianceTypical outcome
Lambda = 0Ordinary least squares valuesNoneHighestOverfitting when features correlate
Small lambdaSlightly shrunkLowReducedOften the best test error
Moderate lambdaClearly shrunk, none exactly zeroModerateLowStable, smooth predictions
Very large lambdaClose to zeroHighVery lowUnderfitting, near-constant output
Effect of the penalty strength on a ridge model

Key parameters and preprocessing for ridge models

The behavior of a ridge model depends on a small set of decisions that matter more than the solver choice. Because the penalty treats every coefficient equally, the scale of each feature directly changes how strongly it is penalized, which makes feature scaling mandatory rather than optional.

SettingTypical valuesWhat it affects
Penalty strength (alpha)Log grid from 0.0001 to 10,000Amount of shrinkage and test error
Feature standardizationZero mean, unit varianceFair penalty across features
Intercept handlingFitted, not penalizedCorrect baseline prediction
SolverCholesky, SVD, conjugate gradient, SAGSpeed and memory on large data
Sample weightsUniform or importance weightsInfluence of individual rows
Settings that control a ridge regression fit

The penalty strength is chosen by cross-validation over a logarithmic grid, since useful values often span several orders of magnitude. Efficient implementations exploit the closed form to compute leave-one-out error almost for free, which makes generalized cross-validation a fast and reliable default. The effective degrees of freedom, the sum of the SVD shrinkage factors, falls smoothly from the number of features toward zero as lambda grows and is a useful diagnostic of model complexity.

Ridge compared with lasso and elastic net

Ridge, lasso and elastic net differ mainly in the shape of the penalty, and that shape decides whether coefficients can become exactly zero. The L2 penalty is smooth, so ridge shrinks every weight proportionally and never performs feature selection. The L1 penalty of lasso has corners at zero, which drives some weights to exactly zero.

MethodPenaltySparse coefficientsCorrelated featuresClosed form
Ridge regressionSum of squared weights (L2)NoShares weight across the groupYes
Lasso regressionSum of absolute weights (L1)YesPicks one, often arbitrarilyNo
Elastic netMix of L1 and L2YesSelects groups togetherNo
Ordinary least squaresNoneNoUnstable, inflated weightsYes, if invertible
Penalized linear models side by side

From a probabilistic angle, ridge is the maximum a posteriori estimate under a Gaussian prior on the coefficients, while lasso corresponds to a Laplace prior. This view explains why ridge spreads credit across correlated predictors: a Gaussian prior prefers many small weights over a few large ones. The same L2 idea appears as weight decay in neural networks and as the regularization term in support vector machines and kernel ridge regression, which applies the kernel trick to the ridge objective.

When to choose ridge regression in practice

Ridge is the right first choice when prediction accuracy matters more than a short list of features and when most inputs are expected to carry some signal. It is fast, deterministic and easy to explain, which makes it a strong baseline before trying random forest or gradient boosting models on tabular data.

SituationRecommended choiceReason
Many correlated predictorsRidge regressionStabilizes and shares weights
More features than samplesRidge or elastic netProblem becomes well posed
Need a sparse, interpretable modelLasso regressionSets weak coefficients to zero
Correlated groups and sparsity wantedElastic netCombines both penalties
Strong nonlinear effectsTree ensembles or kernelsA linear form misses the structure
Decision guide for linear regularization

Typical applications include genomic and spectroscopic data with thousands of correlated measurements, demand forecasting with overlapping calendar and price features, and calibration layers on top of other models. In classification, the same penalty is applied to logistic regression, and the L2-penalized version is the default in most libraries.

Limitations and common mistakes

Ridge regression is robust, but several recurring errors undermine it in production. The most frequent problems come from preprocessing and from misreading the coefficients, not from the algorithm itself.

  • Skipping standardization: features measured in large units are penalized less, so the model silently favors them.
  • Fitting the scaler on all data: computing means and variances before the cross-validation split causes data leakage and optimistic error estimates.
  • Reading coefficients as causal effects: shrinkage biases every weight, and with correlated inputs the split of credit is a modeling choice, not a finding.
  • Expecting feature selection: ridge keeps every feature, so it does not simplify a model with hundreds of irrelevant inputs.
  • Using a coarse linear grid for lambda: good values often differ by factors of ten or more, so a logarithmic search is required.
  • Assuming standard p-values apply: classical confidence intervals do not hold for penalized estimates without specialized corrections.

Ridge also inherits the limits of a linear model. It cannot capture interactions or curvature unless they are engineered into the features, for example through polynomial regression terms, and it remains sensitive to outliers because the squared error loss still dominates the objective.

The idea in brief

Ridge regression is a way of fitting a straight-line style model that refuses to trust any single input too much. Ordinary regression can give huge, unstable weights when several inputs say almost the same thing. Ridge adds a small cost for large weights, so the model spreads its trust more evenly.

Think of a committee where several members repeat each other. Instead of letting one loud voice dominate, ridge asks everyone to speak a little more quietly. The result is a model that is slightly less perfect on old data but more reliable on new data.

A practical scenario

A data team at a regional electricity distributor forecasts next-day load for each substation. Their inputs include 48 hourly temperature readings, humidity, holiday flags and the previous week of load values, about 220 features in total and many of them strongly correlated. An ordinary least squares model fitted on two years of data produced erratic forecasts, with coefficients that flipped sign after every monthly retrain.

The team standardized every feature inside a pipeline, then searched the penalty strength over a logarithmic grid with time-series cross-validation. The selected value of alpha around 30 cut the coefficient swings dramatically and lowered the mean absolute percentage error from about 6.8 percent to 4.9 percent on a held-out season. Because ridge kept every feature, operators could still inspect how temperature at each hour contributed, and retraining became stable enough to automate.