A feedforward neural network (FNN) is an artificial neural network in which information flows in one direction only, from the input layer through one or more hidden layers to the output layer, with no cycles or feedback connections. It is the simplest and most fundamental family of neural networks, and its layers appear as building blocks inside almost every modern architecture in artificial intelligence.
Each layer applies a learned linear transformation followed by a nonlinear activation function. Because the computation is a fixed, acyclic graph, an FNN maps every input to an output in a single forward pass, and it is trained by backpropagation combined with gradient descent.
How a feedforward neural network computes its output
A feedforward network computes its output by applying a sequence of layer functions to the input vector. Each fully connected layer multiplies its input by a weight matrix, adds a bias vector and passes the result through an activation function such as ReLU, GELU, sigmoid or tanh. The output of one layer becomes the input of the next.
Formally, a layer computes h = f(Wx + b), where W holds the weights, b the biases and f the activation. Without the nonlinear activation, stacked layers reduce mathematically to one linear map, so the nonlinearity is what gives depth its value. The final layer uses an activation matched to the task, for example softmax for classification or a linear output for regression.
The table lists the main elements of the forward pass and what each one controls.
| Element | Typical form | What it controls |
|---|---|---|
| Input layer | Feature vector of fixed size | Shape of data the network accepts |
| Hidden layers | Dense layers with activations | Capacity to learn intermediate features |
| Weights and biases | Real-valued parameters | The learned mapping itself |
| Activation function | ReLU, GELU, tanh, sigmoid | Nonlinearity and gradient behavior |
| Output layer | Softmax, sigmoid or linear | Form of the prediction |
Training with backpropagation
A feedforward network learns by minimizing a loss function over training data. After the forward pass produces a prediction, a loss such as cross-entropy or mean squared error measures the error, and backpropagation uses the chain rule to obtain the partial derivative of that loss for each parameter.
Next, a gradient-based method like SGD or Adam nudges every parameter against its gradient so the loss falls. The acyclic structure makes this efficient: gradients flow backward through the same layers exactly once, with no need to unroll loops over time as in recurrent networks.
Several design choices strongly influence whether training succeeds, as the following table shows.
| Choice | Common setting | Effect on training |
|---|---|---|
| Weight initialization | He or Xavier (Glorot) scaling | Keeps activations and gradients stable |
| Activation function | ReLU or GELU in hidden layers | Reduces vanishing gradients |
| Learning rate | About 1e-4 to 1e-2 | Too high diverges, too low stalls |
| Regularization | Dropout, weight decay | Limits overfitting |
| Normalization | Batch or layer normalization | Speeds and stabilizes convergence |
Deep stacks of sigmoid or tanh layers suffer from the vanishing gradient problem, in which gradients shrink layer by layer until early layers stop learning. Residual connections, careful initialization and ReLU-family activations are the standard remedies.
Feedforward networks compared with other architectures
The defining difference between feedforward and other neural networks is the absence of recurrence: an FNN has no memory of previous inputs. The term is broader than multilayer perceptron (MLP), which usually means a fully connected feedforward network; convolutional networks and transformers are also feedforward in the strict sense, because their computation graphs contain no cycles.
The table compares the fully connected FNN with its closest relatives.
| Architecture | Connectivity | Best suited for | Key limitation |
|---|---|---|---|
| Fully connected FNN (MLP) | Every unit linked to the next layer | Tabular data, small fixed inputs | No built-in structure for images or sequences |
| Convolutional neural network | Local, shared-weight filters | Images, spatial signals | Fixed receptive field per layer |
| Recurrent neural network | Cycles carry hidden state over time | Sequential data | Slow, hard-to-parallelize training |
| Transformer | Attention plus position-wise FNN blocks | Language and long sequences | Attention cost grows with sequence length |
Inside every transformer block sits a position-wise feedforward sublayer, typically expanding the hidden dimension about four times before projecting it back. In large language models these sublayers hold roughly two thirds of the parameters, and mixture-of-experts designs replace them with many parallel feedforward experts.
Capacity, depth and the universal approximation theorem
The universal approximation theorem states that a feedforward network with a single hidden layer and a suitable nonlinear activation can approximate any continuous function on a bounded domain to arbitrary accuracy, given enough hidden units. This result guarantees expressive power, but it says nothing about how many units are needed or whether training will find the right weights.
In practice, depth is more efficient than width. Deeper networks can represent many functions with far fewer parameters than a single wide layer, because each layer composes features learned by the previous one. Model size is usually described by the number of layers, the width of each layer and the total parameter count.
- Width sets how many features one layer can represent at once.
- Depth sets how many levels of composition the network can build.
- Parameter count for a dense layer equals inputs times outputs plus one bias per output.
Practical limitations and common mistakes
A plain feedforward network requires fixed-size input and treats every feature independently of position, so it ignores spatial and temporal structure unless that structure is engineered into the features. It is also prone to overfitting on small datasets because dense layers contain many parameters.
The table pairs frequent symptoms with their usual causes and fixes.
| Symptom | Likely cause | Fix |
|---|---|---|
| Loss does not decrease | Learning rate too low or dead ReLU units | Tune learning rate, try leaky ReLU |
| Loss becomes NaN | Learning rate too high, exploding gradients | Lower rate, clip gradients |
| Training accuracy high, test accuracy low | Overfitting | Add dropout, weight decay, more data |
| Slow convergence | Unscaled input features | Standardize inputs to zero mean, unit variance |
A common misconception holds that deep learning always beats classical methods. On tabular data, gradient-boosted decision trees often match or outperform an MLP, so a feedforward network should be benchmarked against strong baselines before it is chosen.
Put simply
A feedforward neural network works like an assembly line. Raw information enters at one end. Each station transforms it a little and passes it on. A finished answer comes out at the other end.
Nothing ever travels backward along the line while it makes a prediction. During training, the network compares its answers with the correct ones. It then adjusts each station slightly. After many rounds, the line turns inputs into reliable answers.
In practice
A regional bank wants to flag loan applications with a high risk of default. Its data team prepares 42 numeric features per applicant, including income, debt-to-income ratio, credit history length and past delinquencies, all standardized to zero mean and unit variance.
The team trains a feedforward network with two hidden layers of 128 and 64 ReLU units, dropout of 0.3 and a single sigmoid output that estimates the probability of default. Trained with Adam and cross-entropy loss on 400,000 historical loans, it is compared against a gradient-boosted tree model on a held-out set.
Both models reach similar discrimination, so the bank deploys an ensemble of the two, which slightly improves ranking quality. Because each prediction needs only one forward pass, the network scores an application in well under a millisecond, fast enough for real-time decisions at the branch counter.