Polynomial regression is a supervised learning method that models the target as a polynomial function of the input features, so a model that is still linear in its coefficients can describe curved relationships. In machine learning, it is the standard first step beyond straight-line fits: the inputs are expanded into powers and interaction terms, and an ordinary linear regression is trained on the expanded features.
The method sits at the heart of the bias-variance tradeoff. A low polynomial degree may miss real curvature, while a high degree can pass through every training point and still predict poorly on new data. Choosing the degree and the regularization strength is therefore the central practical decision.
How polynomial regression expands the inputs
Polynomial regression works by transforming the feature space before fitting a linear model. For a single input x and degree d, the model is y = b0 + b1x + b2x^2 + … + bdx^d. Each power becomes a separate column, and the coefficients are estimated with ordinary least squares, which minimizes the sum of squared residuals.
The model is called linear in the parameters because each coefficient multiplies a fixed, known feature. This property matters: the loss surface is convex, the solution is unique when the design matrix has full rank, and the closed-form normal equations or standard solvers such as QR decomposition apply directly.
With several inputs, the expansion also creates interaction terms such as x1x2 or x1^2×3. The number of generated features grows quickly, as the following table shows for a full expansion that includes all terms up to the given degree.
| Original features | Degree 2 | Degree 3 | Degree 4 |
|---|---|---|---|
| 1 | 3 | 4 | 5 |
| 5 | 21 | 56 | 126 |
| 10 | 66 | 286 | 1,001 |
| 20 | 231 | 1,771 | 10,626 |
The count equals the binomial coefficient C(n + d, d), so feature explosion makes high-degree expansions impractical for wide datasets. This combinatorial growth is a direct instance of the curse of dimensionality.
Key parameters and design choices
A polynomial regression model is defined by a small set of choices that together determine its flexibility and stability. The polynomial degree is the most influential, but scaling, the basis and the penalty matter almost as much in practice.
| Setting | Typical values | Effect on the model |
|---|---|---|
| Degree | 2 to 4 in most applied work | Controls curvature and the risk of overfitting |
| Interaction only | True or false | Keeps cross-products and drops pure powers |
| Feature scaling | Standardization or min-max | Reduces numerical instability of high powers |
| Regularization | None, ridge, lasso or elastic net | Shrinks coefficients and limits variance |
| Basis | Raw powers or orthogonal polynomials | Changes conditioning, not the fitted curve |
Feature scaling is essential because raw powers differ by orders of magnitude. If x ranges up to 1,000, then x^4 reaches 10^12, and the columns become highly correlated. This multicollinearity inflates coefficient variance and makes the individual coefficients almost impossible to interpret, even when predictions remain reasonable.
Orthogonal polynomial bases, such as Legendre polynomials or the orthogonal contrasts used in statistics packages, produce exactly the same fitted values as raw powers of the same degree, but with a much better condition number. They are the preferred choice when the coefficients themselves will be inspected.
How polynomial regression compares with related methods
Polynomial regression is one of several ways to model nonlinear relationships, and its main advantage is simplicity and transparency. Its main weakness is that a global polynomial imposes the same shape on the entire input range, which neighboring methods avoid in different ways.
| Method | How it captures nonlinearity | Main strength | Main weakness |
|---|---|---|---|
| Polynomial regression | Global powers of the inputs | Closed form and interpretable | Unstable at the edges of the data |
| Splines | Piecewise polynomials joined at knots | Local flexibility | Knot placement must be chosen |
| Kernel methods | Implicit feature maps via the kernel trick | Handles very high degrees cheaply | Scales poorly with sample size |
| Decision trees and ensembles | Recursive splits of the input space | Captures interactions automatically | Piecewise constant, poor extrapolation |
| Gaussian process | Prior over smooth functions | Calibrated uncertainty | Cubic cost in the number of samples |
A polynomial kernel in a support vector machine computes the same feature space implicitly, so the kernel trick allows high-degree interactions without materializing thousands of columns. Ridge regression and lasso regression are not alternatives but complements: they are routinely applied on top of the expanded features.
Choosing the degree and fitting the model
The degree should be selected by cross-validation, not by the training error, because training error always decreases as the degree rises. A typical workflow is to scale the features, expand them, fit a regularized linear model and compare validation error across candidate degrees and penalty values within a single pipeline.
| Situation | Recommended choice | Reason |
|---|---|---|
| One input with smooth, gentle curvature | Degree 2 or 3 without penalty | Few parameters and stable fit |
| Several inputs with suspected interactions | Degree 2 with ridge regularization | Captures pairwise effects while limiting variance |
| Many inputs and a sparse true signal | Degree 2 with lasso or elastic net | Selects the useful terms automatically |
| Strongly local behavior or sharp bends | Splines or tree ensembles instead | A global polynomial cannot bend locally |
| Predictions needed outside the data range | Avoid high degrees entirely | Polynomials diverge rapidly beyond the data |
Fitting must happen inside the cross-validation loop: the scaler and the expansion are fit on each training fold only. Fitting them on the full dataset first leaks information from the validation folds and produces optimistic error estimates.
Limitations and common mistakes
The best-known failure mode is Runge’s phenomenon: high-degree polynomials fitted on evenly spaced points oscillate wildly near the boundaries of the interval. The model may look excellent in the middle of the data and produce absurd values at the edges, which is exactly where decisions are often made.
| Symptom | Likely cause | Fix |
|---|---|---|
| Near-zero training error but poor test error | Degree too high, overfitting | Lower the degree or add regularization |
| Huge or unstable coefficients | Unscaled inputs and collinear powers | Standardize features or use an orthogonal basis |
| Wild predictions beyond the observed range | Extrapolation of a global polynomial | Restrict use to the data range or switch methods |
| Systematic pattern in residuals | Degree too low, underfitting | Raise the degree or add relevant interactions |
| Memory errors on wide data | Combinatorial feature explosion | Use interaction-only terms or a kernel method |
- Polynomial regression is not nonlinear regression in the statistical sense; the parameters still enter linearly.
- A high coefficient of determination on training data does not prove the curve is correct.
- Individual coefficients of raw powers are rarely interpretable on their own because they are strongly correlated.
In plain words
Polynomial regression is a way to draw a curved line through data instead of a straight one. It does this by giving the model extra ingredients, such as the square or the cube of each input, and then letting an ordinary straight-line method decide how much of each ingredient to use.
Think of bending a flexible ruler to follow a set of points. A slightly bent ruler follows the general trend. A ruler bent into many waves can touch every point, but it will zigzag in silly ways between and beyond them. The skill lies in choosing just enough bend to follow the real pattern without copying the noise.
A real-world example
A data team at a regional electricity distributor needs to predict daily power demand from outdoor temperature. Demand rises on cold days because of heating and again on hot days because of air conditioning, so a straight line fits badly. The team standardizes temperature, adds squared and cubic terms, and fits a ridge model on three years of daily data.
Five-fold cross-validation compares degrees 1 through 6. Degree 1 leaves a clear U-shaped pattern in the residuals, while degrees above 3 reduce training error but increase validation error. The team selects degree 3 with a light ridge penalty, which cuts the mean absolute error by roughly a third compared with the linear model.
During an unusual heat wave, temperatures exceed anything in the training data. Because analysts know that polynomials extrapolate poorly, they cap the model input at the historical maximum and flag those days for manual review, which prevents an unrealistic demand forecast from reaching the scheduling system.