Elastic net

Also known as: elastic net regularization

On this page 7

Elastic net is a regularized linear modeling method that adds a weighted combination of the L1 (lasso) penalty and the L2 (ridge) penalty to the training loss, so that it can both shrink coefficients and set some of them exactly to zero. In machine learning, elastic net regularization is a standard tool for regression and classification problems with many features, strong correlations between predictors or more features than observations.

The method was proposed by Hui Zou and Trevor Hastie in 2005 to fix two weaknesses of the lasso: its unstable choice among correlated predictors and its limit of selecting at most as many variables as there are samples. Elastic net keeps the sparsity of the lasso while borrowing the stability of ridge regression.

How the elastic net penalty works

Elastic net minimizes the usual model loss plus a penalty term built from two parts. For linear regression the objective is the mean squared error plus lambda * (alpha * ||w||_1 + (1 - alpha) / 2 * ||w||_2^2), where w is the coefficient vector, lambda controls overall strength and alpha sets the L1 to L2 mix.

The L1 component has a corner at zero, which allows the optimizer to push weak coefficients to exactly zero and thereby performs feature selection. The L2 component is smooth and strictly convex, so it spreads weight across correlated predictors and makes the overall problem strictly convex with a unique solution. Together they produce the so-called grouping effect: highly correlated features tend to enter or leave the model together with similar coefficients.

Most implementations fit elastic net with coordinate descent, which updates one coefficient at a time with a soft-thresholding step, and they compute a whole regularization path of solutions for a decreasing sequence of lambda values. Warm starts along this path make fitting hundreds of models only slightly more expensive than fitting one.

The table below summarizes what each element of the objective contributes to the fitted model.

ComponentMathematical formEffect on the model
Data lossSquared error or log lossFits the training data
L1 penaltySum of absolute weightsZeroes weak coefficients, creates sparsity
L2 penaltySum of squared weightsShrinks and stabilizes correlated weights
LambdaNon-negative scalarSets total regularization strength
AlphaValue between 0 and 1Balances sparsity against stability
Components of the elastic net objective

Key hyperparameters and their typical ranges

Elastic net has two hyperparameters that must be tuned together, and their names differ between libraries. In glmnet and the statistics literature, alpha is the mixing weight and lambda the strength; in scikit-learn the same roles are played by l1_ratio and alpha, a frequent source of configuration errors.

The following table lists common settings and how they change model behavior.

SettingTypical search rangeWhat happens at the extremes
Mixing weight0.1 to 0.9, often 0.5Near 1 behaves like lasso, near 0 like ridge
Penalty strengthLog grid of 50 to 100 valuesVery high gives all zeros, very low gives no shrinkage
Convergence tolerance1e-4 to 1e-6Loose values stop early with inexact weights
Maximum iterations1,000 to 10,000Too few iterations trigger convergence warnings
Elastic net hyperparameters in practice

Cross-validation over both parameters is the standard tuning approach. A common practice is to fix a small set of mixing weights, compute the full lambda path for each, and choose the combination with the lowest validation error, or the one-standard-error rule when a sparser and more stable model is preferred.

Elastic net compared with ridge and lasso regression

Elastic net is a generalization of both ridge regression and lasso regression, which appear as its two limiting cases. The choice among the three depends on whether the goal is prediction, interpretation through sparsity, or robustness to correlated inputs.

MethodPenaltyFeature selectionBehavior with correlated features
Ridge regressionL2 onlyNo, all weights stay nonzeroShares weight evenly, stable
Lasso regressionL1 onlyYes, sparse solutionsPicks one feature arbitrarily, unstable
Elastic netL1 plus L2Yes, sparse solutionsKeeps correlated groups together
Ridge, lasso and elastic net side by side

In the p greater than n setting, where features outnumber observations, the lasso can select at most n variables before it saturates. Elastic net has no such hard limit, which matters in genomics, text models and other wide datasets. Ridge regression handles wide data well but never produces a sparse, interpretable model.

When to use elastic net and how to configure it

Elastic net is most valuable when the analyst expects only part of the features to matter and also knows that predictors are correlated. It applies not only to linear regression but also to logistic regression, Poisson regression and Cox models, and to the weights of linear classifiers trained with stochastic gradient descent.

The decision table below maps common data situations to a recommended configuration.

SituationRecommended choiceReason
Many correlated predictors, sparse signalElastic net with mixing weight near 0.5Selects whole groups of related features
Few features, all likely relevantRidge regressionSparsity adds no benefit
Independent features, sparse signalLasso or elastic net near 1Simplest sparse solution
Features far exceed samplesElastic netAvoids the lasso selection limit
Strongly nonlinear relationshipsTree ensembles or kernelsA linear penalty cannot model curvature
Choosing a regularization strategy

Feature scaling is mandatory before fitting, because the penalty treats all coefficients on the same scale. Standardizing each feature to zero mean and unit variance ensures that the penalty reflects predictive importance rather than measurement units. One-hot encoded categories should be handled consistently, ideally with the same scaling choice for every dummy column.

Limitations and common mistakes

Elastic net is a linear method with biased estimates. The penalty deliberately shrinks coefficients toward zero, so their magnitudes are not unbiased effect sizes and standard p-values do not apply. Practitioners who need valid inference must use dedicated post-selection inference techniques or refit an unpenalized model on a separate sample.

  • Skipping standardization makes the penalty favor features measured in large units.
  • Leaking information by scaling or selecting features on the full dataset before cross-validation inflates validation scores.
  • Treating selected features as causal ignores that elastic net optimizes prediction, not causal identification.
  • Confusing parameter names between libraries silently produces a ridge or lasso model instead of the intended mix.
  • Tuning only lambda with a fixed mixing weight can miss a considerably better configuration.

Elastic net also inherits a mild double shrinkage problem: the combined penalties can shrink retained coefficients more than necessary. The original paper proposed rescaling the naive solution by a factor of 1 + lambda2, and some software applies a similar correction, while other packages leave it to the user.

In everyday language

Imagine a team project where a manager must decide which members deserve credit. One strict manager names only a single star and ignores everyone else, even close collaborators who did similar work. Another generous manager gives everybody a small share, even people who contributed nothing.

Elastic net acts like a balanced manager. It removes people who truly added nothing, but when several colleagues worked closely together it credits the whole group instead of picking one at random. The result is a shorter, fairer list of what matters, which makes the model easier to trust and less likely to change wildly when the data changes slightly.

Seen in the real world

A biostatistics team at a university hospital wants to predict how patients with a certain cancer respond to a drug, using the expression levels of about 20,000 genes measured in only 300 patients. Genes in the same biological pathway are strongly correlated, and the team expects that only a few pathways drive the response.

A plain lasso model picks a handful of genes, but the selected genes change almost completely when the team reruns the analysis on bootstrap samples. The analysts standardize all expression values, then fit an elastic net logistic regression with mixing weights of 0.2, 0.5 and 0.8, each with a 100-value lambda path and 10-fold cross-validation nested inside an outer validation split.

The chosen model, with a mixing weight of 0.5, keeps around 120 genes grouped into a few coherent pathways and achieves validation performance similar to the lasso. Its gene selection proves far more stable across resamples, which gives the laboratory a credible shortlist for follow-up experiments.