A neural network, or artificial neural network, is a machine learning model composed of layers of simple computing units (neurons) connected by weighted links. Each unit computes a weighted sum of its inputs, adds a bias and applies a nonlinear activation function; stacking many such layers lets the network represent highly complex functions that map inputs such as pixels, words or sensor readings to outputs such as labels, values or generated text.
Neural networks are the foundation of deep learning and of nearly all modern artificial intelligence systems, from image recognition to large language models. They learn by adjusting millions to trillions of parameters with gradient-based optimization so that their predictions match training data. Architectures such as convolutional networks, recurrent networks and the transformer are specialized neural networks.
How a neural network computes
In a feedforward network, data flows from an input layer through one or more hidden layers to an output layer. A layer multiplies its input vector by a weight matrix, adds a bias vector and applies an activation function elementwise. Without the nonlinearity, any stack of layers would collapse into a single linear transformation; with it, a network with enough units can approximate a very wide class of functions, a result known as the universal approximation theorem.
The output layer is shaped by the task: a softmax over classes for classification, a linear unit for regression, or a distribution over a vocabulary for language generation. The table below summarizes the building blocks.
| Component | Function | Typical choice |
|---|---|---|
| Neuron (unit) | Weighted sum plus bias, then activation | Thousands to millions per model |
| Weights and biases | Learnable parameters | Initialized randomly with scaled variance |
| Activation function | Introduces nonlinearity | ReLU, GELU, sigmoid, tanh |
| Hidden layer | Builds intermediate representations | Few in small models, hundreds in deep ones |
| Output layer | Produces the prediction | Softmax, sigmoid or linear |
| Loss function | Measures prediction error | Cross-entropy, mean squared error |
How neural networks learn
Training minimizes a loss function over examples. For each mini-batch, the network performs a forward pass to compute predictions and loss, then backpropagation applies the chain rule to compute the gradient of the loss with respect to every weight. An optimizer such as stochastic gradient descent or Adam then updates the weights a small step against the gradient. Repeating this over many passes through the data, called epochs, gradually reduces the error.
| Hyperparameter | Typical range | Effect |
|---|---|---|
| Learning rate | About 1e-5 to 1e-1, depending on optimizer | Too high diverges, too low trains slowly |
| Batch size | About 16 to several thousand examples | Gradient noise, memory use, throughput |
| Number of epochs | One to hundreds | Underfitting versus overfitting |
| Weight decay | About 0 to 0.1 | Penalizes large weights to improve generalization |
| Dropout rate | About 0.1 to 0.5 | Randomly disables units to reduce overfitting |
| Depth and width | From a few layers to hundreds | Capacity, compute cost, trainability |
Deep networks once suffered from vanishing and exploding gradients, where gradients shrink or grow exponentially across layers. Practical fixes include ReLU-family activations, careful initialization, normalization layers and residual connections, which add a layer’s input to its output and let very deep models train stably.
Main neural network architectures
Architectures differ in how neurons are connected, which builds assumptions about the data into the model. Choosing an architecture that matches the structure of the data, such as spatial grids, sequences or sets, reduces the amount of data needed to learn well.
| Architecture | Key mechanism | Typical data and tasks |
|---|---|---|
| Multilayer perceptron | Fully connected layers | Tabular data, small prediction tasks |
| Convolutional neural network | Shared local filters over grids | Images, video, spectrograms |
| Recurrent neural network and LSTM | Hidden state carried across time steps | Time series, older speech and text systems |
| Transformer | Attention mechanism across all positions | Language, code, images, multimodal models |
| Autoencoder | Compress then reconstruct input | Representation learning, anomaly detection |
| Graph neural network | Message passing between connected nodes | Molecules, networks, recommendations |
Large modern systems often combine these elements, for example an encoder-decoder architecture built from transformer blocks, or a mixture of experts in which a router activates only a subset of subnetworks for each input to increase capacity without a proportional rise in compute.
Neural networks compared with other machine learning models
Neural networks excel when data is large and unstructured, such as images, audio and text, because they learn useful features automatically. On small or tabular datasets, gradient-boosted decision trees frequently match or beat them with less tuning and better interpretability.
| Situation | Recommended model | Reason |
|---|---|---|
| Millions of labeled images | Convolutional network or vision transformer | Learns visual features directly |
| Natural language understanding or generation | Transformer-based model | Captures long-range context |
| Few thousand rows of tabular data | Gradient-boosted trees | Strong accuracy with little data |
| Strict explainability requirement | Linear model or shallow tree | Transparent decision logic |
| Limited compute at the edge | Small or quantized network | Fits memory and latency limits |
Limitations and common misconceptions
- Not a model of the brain: artificial neurons are loosely inspired by biology but work very differently.
- Data hunger: large networks need large, representative training sets and overfit small ones.
- Opacity: learned weights are hard to interpret, which complicates auditing and debugging.
- Brittleness: performance can drop sharply under distribution shift or adversarial inputs.
- Compute cost: training and inference of large models require substantial hardware and energy.
- Overconfidence: output probabilities are often poorly calibrated without explicit correction.
Put simply
A neural network is a large set of adjustable dials connected in layers. Data goes in one side and an answer comes out the other. At first the dials are set randomly, so the answers are wrong.
During training, the network sees many examples with the correct answers. After each mistake, every dial is nudged slightly in the direction that would have helped. After millions of small corrections, the network gives good answers on examples it has never seen.
In practice
A manufacturer of circuit boards inspects every board with a camera at the end of the line. Rule-based inspection missed subtle solder defects and raised many false alarms. The quality team trained a convolutional neural network on about 50,000 labeled images of good and defective boards, using data augmentation such as small rotations and lighting changes.
The network runs on a small accelerator next to the line and classifies each image in a few milliseconds. Boards it scores as uncertain go to a human inspector. Over the following months, false alarms fell substantially while defect detection improved, and new defect types were handled by retraining with a few hundred additional examples rather than rewriting inspection rules.