Multilayer perceptron

Also known as: MLP

On this page 7

A multilayer perceptron (MLP) is a fully connected feedforward neural network made of an input layer, one or more hidden layers and an output layer, in which every unit applies a nonlinear activation function to a weighted sum of the previous layer’s outputs. Within neural networks and artificial intelligence more broadly, the MLP is the canonical general-purpose architecture and the reference point against which more specialized designs are described.

The MLP extends the single-layer perceptron by stacking layers, which lets it represent nonlinear decision boundaries such as the XOR function. It is trained with backpropagation and gradient descent, and it survives today both as a standalone model for tabular data and as the feedforward block inside every transformer layer.

How a multilayer perceptron computes an output

An MLP computes its output through a sequence of affine transformations and nonlinearities. For each layer, the input vector is multiplied by a weight matrix, a bias vector is added, and an activation function is applied element by element. The output of one layer becomes the input of the next, so information flows strictly forward with no cycles or recurrence.

Formally, layer l computes h = f(W h_prev + b), where W has one row per unit in the layer. Because every unit connects to every unit in the preceding layer, these are called dense or fully connected layers. The number of parameters in one layer equals inputs times outputs plus outputs, which makes parameter count grow quickly with width.

The table below describes each layer type and what it contributes.

LayerFunctionTypical size
Input layerHolds feature values; performs no computationEqual to the number of features
Hidden layerLearns intermediate representationsTens to thousands of units
Output layerProduces predictions in the task formatOne unit per class or target
Layers of a multilayer perceptron

Without nonlinear activations, any stack of layers collapses into a single linear transformation, so depth would add nothing. The nonlinearity is therefore what gives the MLP its expressive power.

Activation functions and output heads

The choice of activation function strongly affects trainability and gradient flow. Early MLPs used sigmoid and tanh, which saturate for large inputs and contribute to the vanishing gradient problem in deeper stacks. Modern MLPs mostly use ReLU or smooth variants such as GELU and SiLU.

The following table compares common hidden-layer activations and their practical trade-offs.

ActivationOutput rangeStrengthWeakness
Sigmoid0 to 1Smooth, probabilistic readingSaturates, slows learning
Tanh-1 to 1Zero-centered outputStill saturates
ReLU0 to infinityCheap, strong gradientsUnits can die at zero
GELUAbout -0.17 to infinitySmooth, standard in transformersSlightly more compute
SwiGLUUnboundedGated, better quality per parameterExtra weight matrix
Common hidden-layer activations in MLPs

The output layer is matched to the task: softmax with cross-entropy loss for multiclass classification, a sigmoid for binary or multilabel problems, and a linear output with mean squared error for regression.

Training an MLP with backpropagation

An MLP learns by minimizing a loss function over training examples. A forward pass produces predictions, the loss measures their error, and backpropagation propagates that error backward, layer by layer, yielding a partial derivative for each weight and bias. Adam, SGD with momentum or a similar optimizer then nudges each parameter against its derivative.

Several settings determine whether training converges well. Weight initialization schemes such as Xavier or He keep activations at a stable scale across layers. Learning rate is usually the most sensitive hyperparameter, and regularization through dropout, weight decay or early stopping limits overfitting.

The table below lists the main design choices and their usual starting values.

SettingCommon starting pointEffect if poorly chosen
Hidden layers2 to 4Too few underfit; too many slow training
Units per layer64 to 1,024Excess width memorizes noise
Learning rate0.001 with AdamDivergence or very slow progress
Batch size32 to 512Noisy or overly smooth updates
Dropout rate0.1 to 0.5Underfitting if set too high
Key training settings for an MLP

MLP compared with other neural architectures

The MLP makes minimal assumptions about input structure, which is both its strength and its weakness. It treats every input feature as independent of position, so it cannot exploit the spatial locality of images or the order of sequences unless that structure is engineered into the features.

The following comparison shows how specialized architectures add inductive bias that the MLP lacks.

ArchitectureConnectivityBuilt-in assumptionTypical data
MLPDense, all-to-allNone beyond smoothnessTabular features, embeddings
Convolutional networkLocal, shared kernelsSpatial localityImages, audio spectrograms
Recurrent networkConnections through timeSequential orderTime series, text
TransformerAttention plus MLP blocksLearned pairwise relationsText, code, multimodal
MLP and related architectures

The universal approximation theorem states that an MLP with one sufficiently wide hidden layer can approximate any continuous function on a bounded domain. The theorem guarantees existence, not learnability: it says nothing about required width, data or training success, which is why deeper and more structured networks often perform better in practice.

Practical limitations of multilayer perceptrons

MLPs are simple to build but have predictable failure modes that practitioners check for early.

  • Unscaled inputs destabilize training; standardize numeric features before fitting.
  • Overfitting on small tables is common; on many tabular benchmarks, gradient-boosted trees still outperform a plain MLP.
  • Parameter inefficiency on images: a dense layer on a 224 by 224 color image needs about 150,000 weights per hidden unit.
  • Poor extrapolation: predictions outside the training range are often unreliable.
  • Opaque reasoning: hidden representations are hard to interpret without dedicated explanation methods.

Explained without jargon

A multilayer perceptron is like an assembly line of decision makers. The first row receives raw facts, such as the size and age of a house. Each worker in the next row looks at everything the previous row passed along, weighs it in their own way, and passes a single opinion forward.

After a few rows, the last worker gives the final answer, such as a price estimate. At first the opinions are random, but after comparing answers with real outcomes thousands of times, every worker adjusts how much attention they pay to each input. The line gradually learns to give useful answers without anyone writing down the rules.

A typical use case

A regional utility wants to forecast hourly electricity demand for the next day. Its data team builds a table of features for each hour: temperature, humidity, hour of day, day of week, holiday flags and the demand recorded 24 and 168 hours earlier. After standardizing the inputs, they train an MLP with three hidden layers of 256 ReLU units, dropout of 0.2 and a single linear output.

Trained on four years of history and validated on the most recent year, the model captures interactions that a linear regression missed, such as heat that raises demand more on weekday afternoons. The team compares it against a gradient-boosted tree baseline, keeps whichever performs better each season, and uses the forecasts to schedule generation and reduce reserve costs.