Universal approximation theorem

On this page 7

The universal approximation theorem is a mathematical result stating that a feedforward neural network with a single hidden layer and a suitable nonlinear activation function can approximate any continuous function on a compact domain to any desired accuracy, provided the hidden layer contains enough neurons. In the study of neural networks within artificial intelligence, it is the standard justification that these models are expressive enough in principle to represent almost any input-output mapping.

The theorem is an existence result. It guarantees that suitable weights exist, but it says nothing about how many neurons are needed in practice, how to find those weights by training or whether the resulting network will generalize to new data.

What the universal approximation theorem states

In its classic form, the theorem considers a continuous target function f defined on a compact subset of n-dimensional real space, such as the unit cube. For any tolerance epsilon greater than zero, there exists a network g with one hidden layer of N units such that the maximum difference between f(x) and g(x) over the whole domain is smaller than epsilon.

The approximating network has the form of a weighted sum of activated affine functions: each hidden unit computes an activation of a weighted input plus a bias, and a linear output layer adds the results. The theorem says that the set of all such sums is dense in the space of continuous functions under the uniform norm.

The table below separates the assumptions of the classic statement from what it concludes.

ElementClassic requirementWhy it matters
Target functionContinuousDiscontinuous targets need other error measures
Input domainCompact, closed and boundedGuarantees hold only inside that region
ArchitectureOne hidden layer, linear outputWidth, not depth, supplies the power
ActivationNon-polynomial, such as sigmoidPolynomial activations fail the theorem
Error measureUniform (supremum) normBounds the worst-case error everywhere
ConclusionSuitable weights existNo promise about learning them
Assumptions and conclusions of the classic theorem

Key versions and historical results

The theorem is a family of related results rather than a single proof. George Cybenko proved the version for sigmoidal activations in 1989, and Kurt Hornik, Maxwell Stinchcombe and Halbert White published a general proof for multilayer feedforward networks in the same year. In 1993, Moshe Leshno and coauthors showed that a one-hidden-layer network is universal if and only if the activation is not a polynomial, which covers ReLU.

Later work shifted attention from width to depth. Deep narrow networks with bounded width can also be universal, and depth separation results show that some functions require exponentially more neurons in a shallow network than in a deeper one.

ResultSettingMain contribution
Cybenko (1989)One hidden layer, sigmoidFirst widely cited proof
Hornik, Stinchcombe and White (1989)General squashing functionsUniversality for broad activation classes
Leshno and coauthors (1993)Any continuous activationNon-polynomial condition is necessary and sufficient
Barron (1993)Functions with bounded Fourier momentError rate that avoids the curse of dimensionality
Bounded-width results (2017 onward)Deep networks of limited widthDepth alone can provide universality
Selected versions of the universal approximation theorem

The Kolmogorov-Arnold representation theorem, which shows that multivariate continuous functions can be written using sums and compositions of univariate functions, is an older related result. It inspired Kolmogorov-Arnold networks, but it is an exact representation theorem rather than an approximation guarantee of the same kind.

Why nonlinearity makes approximation possible

The intuition is that each hidden unit contributes a simple building block, and enough blocks can trace any shape. With sigmoid activations, two units with large weights and opposite signs form an approximate step or bump; many bumps placed side by side build a staircase that follows the target curve as closely as desired.

With ReLU, each unit adds a hinge, so the network output is a piecewise linear function. More units mean more linear pieces and a finer fit. A purely linear activation, by contrast, collapses any number of layers into a single linear map, which is why a nonlinear activation function is essential.

What the theorem does not guarantee

The most common misconception is that the theorem explains why neural networks work well in practice. It addresses only representational capacity, and several practical questions remain outside its scope.

MisreadingActual positionPractical consequence
Networks can learn anythingWeights exist, but training may not find themOptimization by backpropagation can stall
A small network sufficesRequired width can grow exponentiallyShallow models may be impractically large
Good fit means good predictionsTraining fit does not imply generalizationOverfitting remains a central risk
It holds on all inputsOnly on a compact domainExtrapolation outside training range is unreliable
Depth is unnecessaryDepth can be exponentially more efficientDeep architectures dominate in practice
Common misreadings and the actual position

Generalization depends on the amount and quality of training data, on regularization such as dropout and on the inductive bias of the architecture. Convolutional and transformer models succeed because their structure matches the data, not merely because universality holds for every multilayer perceptron.

How practitioners use the theorem

In engineering practice, the theorem serves as a sanity check rather than a design rule. It confirms that a multilayer perceptron is a valid general-purpose function approximator, which supports its use as a component in regression models, value functions in reinforcement learning and physics-informed surrogates.

  • Choosing an activation: any standard non-polynomial activation, including ReLU, GELU and tanh, satisfies the condition, so the choice rests on training behavior.
  • Sizing a network: the theorem gives no width formula; size is set empirically by validation error.
  • Interpreting failures: poor results usually point to optimization, data or regularization problems rather than a lack of expressive power.
  • Bounding inputs: because guarantees apply on a compact domain, inputs are normalized and the model is not trusted far outside the training range.

A plain-language summary

The universal approximation theorem says that a neural network with enough simple units can copy the shape of almost any smooth curve or surface as closely as needed.

Think of building a curved wall from small straight bricks. With a few bricks the wall looks jagged, but with thousands of tiny bricks it follows the curve almost perfectly. The theorem promises that the right bricks exist. It does not tell anyone how many bricks are needed or how to place them, and that practical work is what training has to solve.

An example from practice

An engineering team in a wind turbine company needs a fast replacement for a fluid simulation that takes about 40 minutes per run. The simulation maps eight inputs, including wind speed, blade pitch and air density, to a predicted power output. Because the mapping is continuous over bounded operating ranges, the theorem confirms that a multilayer perceptron can, in principle, approximate it.

The team generates 20,000 simulation results, normalizes every input to the operating range and trains a network with three hidden layers of 128 ReLU units. A single-hidden-layer model of similar size fits less accurately, which reflects the efficiency advantage of depth. The final surrogate predicts power within about 1 percent of the simulator on held-out cases in milliseconds, and the team restricts its use to the trained input ranges, where the approximation is reliable.