Polynomial regression

On this page 7

Polynomial regression is a supervised learning method that models the target as a polynomial function of the input features, so a model that is still linear in its coefficients can describe curved relationships. In machine learning, it is the standard first step beyond straight-line fits: the inputs are expanded into powers and interaction terms, and an ordinary linear regression is trained on the expanded features.

The method sits at the heart of the bias-variance tradeoff. A low polynomial degree may miss real curvature, while a high degree can pass through every training point and still predict poorly on new data. Choosing the degree and the regularization strength is therefore the central practical decision.

How polynomial regression expands the inputs

Polynomial regression works by transforming the feature space before fitting a linear model. For a single input x and degree d, the model is y = b0 + b1x + b2x^2 + … + bdx^d. Each power becomes a separate column, and the coefficients are estimated with ordinary least squares, which minimizes the sum of squared residuals.

The model is called linear in the parameters because each coefficient multiplies a fixed, known feature. This property matters: the loss surface is convex, the solution is unique when the design matrix has full rank, and the closed-form normal equations or standard solvers such as QR decomposition apply directly.

With several inputs, the expansion also creates interaction terms such as x1x2 or x1^2×3. The number of generated features grows quickly, as the following table shows for a full expansion that includes all terms up to the given degree.

Original featuresDegree 2Degree 3Degree 4
1345
52156126
10662861,001
202311,77110,626
Number of features after a full polynomial expansion (including the constant term)

The count equals the binomial coefficient C(n + d, d), so feature explosion makes high-degree expansions impractical for wide datasets. This combinatorial growth is a direct instance of the curse of dimensionality.

Key parameters and design choices

A polynomial regression model is defined by a small set of choices that together determine its flexibility and stability. The polynomial degree is the most influential, but scaling, the basis and the penalty matter almost as much in practice.

SettingTypical valuesEffect on the model
Degree2 to 4 in most applied workControls curvature and the risk of overfitting
Interaction onlyTrue or falseKeeps cross-products and drops pure powers
Feature scalingStandardization or min-maxReduces numerical instability of high powers
RegularizationNone, ridge, lasso or elastic netShrinks coefficients and limits variance
BasisRaw powers or orthogonal polynomialsChanges conditioning, not the fitted curve
Main settings and what they control

Feature scaling is essential because raw powers differ by orders of magnitude. If x ranges up to 1,000, then x^4 reaches 10^12, and the columns become highly correlated. This multicollinearity inflates coefficient variance and makes the individual coefficients almost impossible to interpret, even when predictions remain reasonable.

Orthogonal polynomial bases, such as Legendre polynomials or the orthogonal contrasts used in statistics packages, produce exactly the same fitted values as raw powers of the same degree, but with a much better condition number. They are the preferred choice when the coefficients themselves will be inspected.

Polynomial regression is one of several ways to model nonlinear relationships, and its main advantage is simplicity and transparency. Its main weakness is that a global polynomial imposes the same shape on the entire input range, which neighboring methods avoid in different ways.

MethodHow it captures nonlinearityMain strengthMain weakness
Polynomial regressionGlobal powers of the inputsClosed form and interpretableUnstable at the edges of the data
SplinesPiecewise polynomials joined at knotsLocal flexibilityKnot placement must be chosen
Kernel methodsImplicit feature maps via the kernel trickHandles very high degrees cheaplyScales poorly with sample size
Decision trees and ensemblesRecursive splits of the input spaceCaptures interactions automaticallyPiecewise constant, poor extrapolation
Gaussian processPrior over smooth functionsCalibrated uncertaintyCubic cost in the number of samples
Polynomial regression and alternative nonlinear approaches

A polynomial kernel in a support vector machine computes the same feature space implicitly, so the kernel trick allows high-degree interactions without materializing thousands of columns. Ridge regression and lasso regression are not alternatives but complements: they are routinely applied on top of the expanded features.

Choosing the degree and fitting the model

The degree should be selected by cross-validation, not by the training error, because training error always decreases as the degree rises. A typical workflow is to scale the features, expand them, fit a regularized linear model and compare validation error across candidate degrees and penalty values within a single pipeline.

SituationRecommended choiceReason
One input with smooth, gentle curvatureDegree 2 or 3 without penaltyFew parameters and stable fit
Several inputs with suspected interactionsDegree 2 with ridge regularizationCaptures pairwise effects while limiting variance
Many inputs and a sparse true signalDegree 2 with lasso or elastic netSelects the useful terms automatically
Strongly local behavior or sharp bendsSplines or tree ensembles insteadA global polynomial cannot bend locally
Predictions needed outside the data rangeAvoid high degrees entirelyPolynomials diverge rapidly beyond the data
Decision guide for practical situations

Fitting must happen inside the cross-validation loop: the scaler and the expansion are fit on each training fold only. Fitting them on the full dataset first leaks information from the validation folds and produces optimistic error estimates.

Limitations and common mistakes

The best-known failure mode is Runge’s phenomenon: high-degree polynomials fitted on evenly spaced points oscillate wildly near the boundaries of the interval. The model may look excellent in the middle of the data and produce absurd values at the edges, which is exactly where decisions are often made.

SymptomLikely causeFix
Near-zero training error but poor test errorDegree too high, overfittingLower the degree or add regularization
Huge or unstable coefficientsUnscaled inputs and collinear powersStandardize features or use an orthogonal basis
Wild predictions beyond the observed rangeExtrapolation of a global polynomialRestrict use to the data range or switch methods
Systematic pattern in residualsDegree too low, underfittingRaise the degree or add relevant interactions
Memory errors on wide dataCombinatorial feature explosionUse interaction-only terms or a kernel method
Symptoms, causes and fixes
  • Polynomial regression is not nonlinear regression in the statistical sense; the parameters still enter linearly.
  • A high coefficient of determination on training data does not prove the curve is correct.
  • Individual coefficients of raw powers are rarely interpretable on their own because they are strongly correlated.

In plain words

Polynomial regression is a way to draw a curved line through data instead of a straight one. It does this by giving the model extra ingredients, such as the square or the cube of each input, and then letting an ordinary straight-line method decide how much of each ingredient to use.

Think of bending a flexible ruler to follow a set of points. A slightly bent ruler follows the general trend. A ruler bent into many waves can touch every point, but it will zigzag in silly ways between and beyond them. The skill lies in choosing just enough bend to follow the real pattern without copying the noise.

A real-world example

A data team at a regional electricity distributor needs to predict daily power demand from outdoor temperature. Demand rises on cold days because of heating and again on hot days because of air conditioning, so a straight line fits badly. The team standardizes temperature, adds squared and cubic terms, and fits a ridge model on three years of daily data.

Five-fold cross-validation compares degrees 1 through 6. Degree 1 leaves a clear U-shaped pattern in the residuals, while degrees above 3 reduce training error but increase validation error. The team selects degree 3 with a light ridge penalty, which cuts the mean absolute error by roughly a third compared with the linear model.

During an unusual heat wave, temperatures exceed anything in the training data. Because analysts know that polynomials extrapolate poorly, they cap the model input at the historical maximum and flag those days for manual review, which prevents an unrealistic demand forecast from reaching the scheduling system.