Lasso regression

Also known as: LASSO · Least absolute shrinkage and selection operator

On this page 7

Lasso regression is a linear regression method that adds an L1 penalty on the absolute size of the coefficients to the ordinary least squares loss, so that the model both shrinks its weights and sets many of them exactly to zero. The name LASSO stands for least absolute shrinkage and selection operator, introduced by Robert Tibshirani in 1996.

In machine learning, lasso regression is valued because it performs regularization and feature selection in one step. It produces sparse, interpretable models when only a fraction of the candidate inputs truly matter, and it remains usable when the number of features approaches or exceeds the number of observations.

How the L1 penalty produces sparse coefficients

Lasso regression minimizes the residual sum of squares plus lambda times the sum of absolute coefficient values. Written compactly, the objective is (1/2n)·||y − Xβ||² + λ·||β||₁, where lambda (λ) controls penalty strength and the intercept is normally left unpenalized.

The sparsity comes from the geometry of the L1 constraint. The set of coefficient vectors with a fixed L1 norm forms a diamond in two dimensions and a cross-polytope in higher dimensions, with sharp corners lying on the axes. The elliptical contours of the squared error usually touch this region at a corner, and a corner means that one or more coefficients equal zero.

Algebraically, the same effect appears as soft thresholding. For standardized, uncorrelated features, each lasso coefficient equals the least squares estimate moved toward zero by λ, and any estimate whose magnitude is smaller than λ becomes exactly zero. The following table shows how the penalty strength changes the fitted model.

Lambda settingNonzero coefficientsBiasVarianceTypical outcome
λ = 0All featuresLowestHighestOrdinary least squares fit
Small λMost featuresLowModerateMild shrinkage, little selection
Moderate λA compact subsetModerateLowUsually the best test error
Large λVery few or noneHighVery lowUnderfit, intercept-only model
Effect of the penalty strength on a lasso model

How lasso regression is fitted

The L1 penalty is not differentiable at zero, so lasso regression has no closed-form solution and cannot use plain gradient descent without modification. Several specialized solvers handle this kink efficiently.

AlgorithmCore ideaBest suited for
Coordinate descentUpdates one coefficient at a time by soft thresholdingDefault in most libraries, large sparse data
Least angle regressionTraces the full path as features enter piecewise linearlySmall to medium data, exact path
Proximal gradient (ISTA, FISTA)Gradient step followed by a thresholding stepVery large or streaming problems
ADMMSplits the loss and penalty into separate subproblemsDistributed and constrained settings
Common algorithms for solving the lasso problem

In practice, solvers compute a regularization path over a decreasing grid of lambda values, using each solution as a warm start for the next. This makes it cheap to evaluate dozens of penalty levels, and k-fold cross-validation then selects the value with the lowest validation error or, more conservatively, the largest λ within one standard error of that minimum.

Lasso compared with ridge regression and elastic net

Lasso regression is one of three closely related penalized linear models, and the choice among them depends on whether sparsity or stability matters more. Ridge regression uses an L2 penalty that shrinks all coefficients smoothly but never sets them to zero, while elastic net mixes both penalties.

PropertyLasso regressionRidge regressionElastic net
Penalty termL1, sum of absolute valuesL2, sum of squaresWeighted L1 plus L2
Exact zero coefficientsYesNoYes
Correlated featuresPicks one, often arbitrarilySpreads weight evenlySelects groups together
Features above sample sizeAt most n selectedAll retainedNo hard limit
Closed-form solutionNoYesNo
Main strengthSparse, readable modelStable predictionsSparsity with grouping
Penalized linear models side by side

Unlike classic stepwise feature selection, lasso makes selection a continuous, convex optimization problem, so small changes in data cause gradual rather than abrupt changes in the chosen subset. Compared with tree-based feature importance from a random forest or gradient boosting, lasso captures only linear, additive effects unless interactions or transformed features are added explicitly.

Applying lasso regression in practice

Correct preprocessing is essential because the penalty treats every coefficient on the same scale. Feature scaling to zero mean and unit variance is mandatory; otherwise a feature measured in large units receives a smaller coefficient and is penalized less than an equally useful feature measured in small units.

  • Standardize numeric inputs and fit the scaler on training folds only, to avoid leakage.
  • Encode categorical variables with one-hot encoding; consider the group lasso when all levels of a category should enter or leave together.
  • Tune lambda with cross-validation and report the selected value along with the number of retained features.
  • Refit or debias when needed: the relaxed lasso refits ordinary least squares on the selected features to reduce shrinkage bias.
  • For classification, the same L1 penalty applies to logistic regression, giving sparse classifiers.

The decision table below summarizes when lasso regression is a sensible first choice and when another method fits the problem better.

SituationRecommended choiceReason
Many features, few expected to matterLasso regressionSelects a small, explainable subset
Groups of strongly correlated predictorsElastic netKeeps correlated features together
All features plausibly contributeRidge regressionShrinks without discarding signal
Strong nonlinear interactionsGradient boostingLearns interactions automatically
Valid p-values required after selectionPost-selection inferenceNaive tests are biased after lasso
When to choose lasso regression

Limitations and common mistakes with lasso

Lasso regression is powerful but has well-documented weaknesses that experienced practitioners account for. The most important is instability under correlation: when two predictors carry nearly the same information, lasso tends to keep one and drop the other, and the chosen one can change between bootstrap samples.

  • Shrinkage bias: large true effects are also pulled toward zero, so raw lasso coefficients understate them.
  • Selection consistency is conditional: lasso recovers the true feature set only under strict conditions such as the irrepresentable condition.
  • Selected is not causal: a zero coefficient means the feature was not needed for prediction, not that it has no effect.
  • Naive inference fails: confidence intervals computed after selection on the same data are too narrow.
  • Forgetting to scale silently changes which features survive the penalty.

Remedies include stability selection, which keeps only features chosen across many subsamples, the adaptive lasso, which uses feature-specific weights to reduce bias, and elastic net when correlation is the main concern.

The idea in brief

Imagine fitting a prediction formula that may use hundreds of possible ingredients. Lasso regression charges a fee for every unit of weight the formula gives to an ingredient. Ingredients that help only a little are not worth the fee, so their weight drops to exactly zero and they leave the recipe.

The result is a shorter, simpler formula that keeps only the inputs that really earn their place. A single dial, called lambda, sets how expensive the fee is: a low fee keeps almost everything, and a high fee keeps almost nothing. The best setting is found by testing the formula on data it has not seen.

A practical scenario

A hospital analytics team wants to predict the length of stay for surgical patients from 420 candidate variables: laboratory values, vital signs, diagnosis codes encoded as indicators and demographic data, drawn from about 6,000 past admissions. Many variables overlap, and clinicians insist that any model be short enough to review line by line.

The team standardizes the inputs, fits lasso regression on a log-transformed length of stay and runs 10-fold cross-validation over 100 lambda values. Using the one-standard-error rule, the selected model keeps 27 of the 420 variables, including albumin, age, the American Society of Anesthesiologists class and emergency admission status.

On a held-out year of admissions, the sparse model matches the accuracy of a full ridge model within about 2 percent of mean absolute error. Because two correlated kidney markers swapped places across folds, the team adds stability selection before deployment. The outcome is a compact, auditable model that clinicians accept for discharge planning.