Weight initialization is the choice of starting values for the trainable parameters of a neural network before gradient-based training begins. In neural networks, and in artificial intelligence more broadly, it determines whether activations and gradients keep a stable scale across layers, which directly affects whether a deep model trains quickly, slowly or not at all.
A good scheme breaks symmetry between neurons, keeps the variance of signals roughly constant from layer to layer and matches the chosen activation function. Standard methods such as Xavier (Glorot) initialization and He (Kaiming) initialization derive their scale from the number of inputs and outputs of each layer.
Why the starting weights matter
Starting weights matter because each layer multiplies its inputs by a weight matrix, so small scale errors compound exponentially with depth. If the weights are slightly too large, activations and gradients grow layer by layer and explode; if they are slightly too small, they shrink toward zero and vanish, which stalls learning in early layers.
Initialization also has to break symmetry. If every weight in a layer starts with the same value, all neurons in that layer compute identical outputs and receive identical gradient updates, so they never learn different features. Random sampling from a carefully scaled distribution solves both problems at once.
The table below shows the typical symptoms of each failure mode.
| Initial scale | Effect on signals | Observed symptom |
|---|---|---|
| All zeros or constant | Identical neurons, no symmetry breaking | Layers behave like a single unit |
| Too small | Activations and gradients shrink with depth | Loss barely moves, early layers stop learning |
| Too large | Activations saturate or grow without bound | Loss becomes NaN or oscillates wildly |
| Variance-preserving | Signal scale stays roughly constant | Smooth, fast early convergence |
Main weight initialization schemes
Modern schemes set the weight variance from layer dimensions so that the variance of outputs, or of backpropagated gradients, matches that of inputs. Here, fan-in is the number of inputs to a neuron and fan-out is the number of outputs, and both are counted per kernel position in convolutional layers.
| Method | Weight variance | Designed for |
|---|---|---|
| Xavier (Glorot) | 2 / (fan-in + fan-out) | Tanh, sigmoid and linear layers |
| He (Kaiming) | 2 / fan-in | ReLU and its variants |
| LeCun | 1 / fan-in | SELU and self-normalizing networks |
| Orthogonal | Orthogonal matrix with chosen gain | Recurrent and very deep networks |
| Scaled residual | Base variance divided by depth factor | Deep residual networks and Transformers |
Each rule can be applied with a normal or uniform distribution of the same variance; a uniform version samples from plus or minus the square root of three times the variance. Biases are usually initialized to zero, with exceptions such as a positive forget-gate bias in LSTM cells, often set to 1, to help early memory retention.
Matching initialization to architecture and activation
The correct scheme depends mainly on the activation function and the network topology. He initialization doubles the Xavier variance because ReLU discards roughly half of its inputs, which halves the signal variance; using Xavier with deep ReLU networks therefore lets activations fade.
Normalization layers and residual connections change the picture. Batch normalization and layer normalization rescale activations during training, which makes networks far more tolerant of imperfect initial scales. Residual networks add many branches together, so practitioners often scale down the last layer of each residual branch, sometimes to zero, so that each block starts close to an identity mapping.
| Architecture | Recommended starting point | Reason |
|---|---|---|
| Multilayer perceptron with ReLU | He normal, zero biases | Compensates for ReLU discarding negatives |
| Convolutional neural network | He normal with fan-in mode | Keeps feature map variance stable |
| Recurrent network or LSTM | Orthogonal recurrent weights, forget bias 1 | Limits gradient growth over time steps |
| Transformer | Small normal std, scaled residual outputs | Stabilizes deep stacks with layer normalization |
| Network with SELU | LeCun normal | Required for self-normalizing behavior |
Transfer learning offers a different path entirely: starting from pretrained weights replaces random initialization for most layers, and only newly added heads need a fresh scheme.
Very large language models add further refinements. Some training recipes use a fixed small standard deviation of about 0.02 for most matrices, while others scale it with the inverse square root of model width. Parameterization schemes such as maximal update parameterization go further and tie initialization to learning-rate scaling, so hyperparameters tuned on a small model transfer to a much larger one with fewer costly experiments.
Common mistakes and practical diagnostics
Most initialization problems are detectable in the first few hundred training steps by monitoring per-layer activation statistics and gradient norms. A healthy network shows activation standard deviations of roughly similar magnitude across layers, rather than a steady rise or fall with depth.
- Mismatched scheme and activation, such as Xavier with very deep ReLU stacks, causes gradual signal decay.
- Overriding framework defaults unknowingly: libraries differ in their default schemes, so explicit initialization improves reproducibility.
- Initializing embeddings too large in language models can destabilize early attention scores.
- Ignoring random seeds makes results hard to reproduce, because different draws can change final accuracy noticeably in small models.
- Relying on initialization alone in very deep models; normalization, residual connections and learning-rate warmup are usually needed as well.
When training diverges early, a practical sequence is to check for NaN values, inspect per-layer gradient norms, reduce the initial scale of the final layers and add a short learning-rate warmup before changing the architecture.
The short version
Before a neural network learns anything, all of its internal settings need starting values. Weight initialization is how those first values are chosen.
Picture a long line of people passing a message by whispering. If each person speaks too loudly, the message becomes a shout; if each speaks too softly, it fades away. Good initialization makes every layer pass the signal at the right volume, so learning can start smoothly instead of stalling or blowing up.
Where it makes a difference
A machine learning engineer trains a 50-layer convolutional network for defect detection on factory images. With the library’s Xavier uniform setting and plain ReLU layers without normalization, the training loss stays almost flat for the first two epochs, and per-layer statistics show activation standard deviations dropping by several orders of magnitude between the first and last layers.
Switching to He normal initialization in fan-in mode, with zero biases, keeps activation scales within a narrow band across all layers. The loss starts falling in the first epoch, and the model reaches its target validation accuracy in about half the number of epochs. The engineer documents the scheme and random seed in the training configuration so later experiments remain comparable.