Feedforward neural network

Also known as: FNN

On this page 7

A feedforward neural network (FNN) is an artificial neural network in which information flows in one direction only, from the input layer through one or more hidden layers to the output layer, with no cycles or feedback connections. It is the simplest and most fundamental family of neural networks, and its layers appear as building blocks inside almost every modern architecture in artificial intelligence.

Each layer applies a learned linear transformation followed by a nonlinear activation function. Because the computation is a fixed, acyclic graph, an FNN maps every input to an output in a single forward pass, and it is trained by backpropagation combined with gradient descent.

How a feedforward neural network computes its output

A feedforward network computes its output by applying a sequence of layer functions to the input vector. Each fully connected layer multiplies its input by a weight matrix, adds a bias vector and passes the result through an activation function such as ReLU, GELU, sigmoid or tanh. The output of one layer becomes the input of the next.

Formally, a layer computes h = f(Wx + b), where W holds the weights, b the biases and f the activation. Without the nonlinear activation, stacked layers reduce mathematically to one linear map, so the nonlinearity is what gives depth its value. The final layer uses an activation matched to the task, for example softmax for classification or a linear output for regression.

The table lists the main elements of the forward pass and what each one controls.

ElementTypical formWhat it controls
Input layerFeature vector of fixed sizeShape of data the network accepts
Hidden layersDense layers with activationsCapacity to learn intermediate features
Weights and biasesReal-valued parametersThe learned mapping itself
Activation functionReLU, GELU, tanh, sigmoidNonlinearity and gradient behavior
Output layerSoftmax, sigmoid or linearForm of the prediction
Elements of the forward pass

Training with backpropagation

A feedforward network learns by minimizing a loss function over training data. After the forward pass produces a prediction, a loss such as cross-entropy or mean squared error measures the error, and backpropagation uses the chain rule to obtain the partial derivative of that loss for each parameter.

Next, a gradient-based method like SGD or Adam nudges every parameter against its gradient so the loss falls. The acyclic structure makes this efficient: gradients flow backward through the same layers exactly once, with no need to unroll loops over time as in recurrent networks.

Several design choices strongly influence whether training succeeds, as the following table shows.

ChoiceCommon settingEffect on training
Weight initializationHe or Xavier (Glorot) scalingKeeps activations and gradients stable
Activation functionReLU or GELU in hidden layersReduces vanishing gradients
Learning rateAbout 1e-4 to 1e-2Too high diverges, too low stalls
RegularizationDropout, weight decayLimits overfitting
NormalizationBatch or layer normalizationSpeeds and stabilizes convergence
Training choices and their effects

Deep stacks of sigmoid or tanh layers suffer from the vanishing gradient problem, in which gradients shrink layer by layer until early layers stop learning. Residual connections, careful initialization and ReLU-family activations are the standard remedies.

Feedforward networks compared with other architectures

The defining difference between feedforward and other neural networks is the absence of recurrence: an FNN has no memory of previous inputs. The term is broader than multilayer perceptron (MLP), which usually means a fully connected feedforward network; convolutional networks and transformers are also feedforward in the strict sense, because their computation graphs contain no cycles.

The table compares the fully connected FNN with its closest relatives.

ArchitectureConnectivityBest suited forKey limitation
Fully connected FNN (MLP)Every unit linked to the next layerTabular data, small fixed inputsNo built-in structure for images or sequences
Convolutional neural networkLocal, shared-weight filtersImages, spatial signalsFixed receptive field per layer
Recurrent neural networkCycles carry hidden state over timeSequential dataSlow, hard-to-parallelize training
TransformerAttention plus position-wise FNN blocksLanguage and long sequencesAttention cost grows with sequence length
Feedforward network and related architectures

Inside every transformer block sits a position-wise feedforward sublayer, typically expanding the hidden dimension about four times before projecting it back. In large language models these sublayers hold roughly two thirds of the parameters, and mixture-of-experts designs replace them with many parallel feedforward experts.

Capacity, depth and the universal approximation theorem

The universal approximation theorem states that a feedforward network with a single hidden layer and a suitable nonlinear activation can approximate any continuous function on a bounded domain to arbitrary accuracy, given enough hidden units. This result guarantees expressive power, but it says nothing about how many units are needed or whether training will find the right weights.

In practice, depth is more efficient than width. Deeper networks can represent many functions with far fewer parameters than a single wide layer, because each layer composes features learned by the previous one. Model size is usually described by the number of layers, the width of each layer and the total parameter count.

  • Width sets how many features one layer can represent at once.
  • Depth sets how many levels of composition the network can build.
  • Parameter count for a dense layer equals inputs times outputs plus one bias per output.

Practical limitations and common mistakes

A plain feedforward network requires fixed-size input and treats every feature independently of position, so it ignores spatial and temporal structure unless that structure is engineered into the features. It is also prone to overfitting on small datasets because dense layers contain many parameters.

The table pairs frequent symptoms with their usual causes and fixes.

SymptomLikely causeFix
Loss does not decreaseLearning rate too low or dead ReLU unitsTune learning rate, try leaky ReLU
Loss becomes NaNLearning rate too high, exploding gradientsLower rate, clip gradients
Training accuracy high, test accuracy lowOverfittingAdd dropout, weight decay, more data
Slow convergenceUnscaled input featuresStandardize inputs to zero mean, unit variance
Typical problems when training feedforward networks

A common misconception holds that deep learning always beats classical methods. On tabular data, gradient-boosted decision trees often match or outperform an MLP, so a feedforward network should be benchmarked against strong baselines before it is chosen.

Put simply

A feedforward neural network works like an assembly line. Raw information enters at one end. Each station transforms it a little and passes it on. A finished answer comes out at the other end.

Nothing ever travels backward along the line while it makes a prediction. During training, the network compares its answers with the correct ones. It then adjusts each station slightly. After many rounds, the line turns inputs into reliable answers.

In practice

A regional bank wants to flag loan applications with a high risk of default. Its data team prepares 42 numeric features per applicant, including income, debt-to-income ratio, credit history length and past delinquencies, all standardized to zero mean and unit variance.

The team trains a feedforward network with two hidden layers of 128 and 64 ReLU units, dropout of 0.3 and a single sigmoid output that estimates the probability of default. Trained with Adam and cross-entropy loss on 400,000 historical loans, it is compared against a gradient-boosted tree model on a held-out set.

Both models reach similar discrimination, so the bank deploys an ensemble of the two, which slightly improves ranking quality. Because each prediction needs only one forward pass, the network scores an application in well under a millisecond, fast enough for real-time decisions at the branch counter.