Weight initialization

On this page 6

Weight initialization is the choice of starting values for the trainable parameters of a neural network before gradient-based training begins. In neural networks, and in artificial intelligence more broadly, it determines whether activations and gradients keep a stable scale across layers, which directly affects whether a deep model trains quickly, slowly or not at all.

A good scheme breaks symmetry between neurons, keeps the variance of signals roughly constant from layer to layer and matches the chosen activation function. Standard methods such as Xavier (Glorot) initialization and He (Kaiming) initialization derive their scale from the number of inputs and outputs of each layer.

Why the starting weights matter

Starting weights matter because each layer multiplies its inputs by a weight matrix, so small scale errors compound exponentially with depth. If the weights are slightly too large, activations and gradients grow layer by layer and explode; if they are slightly too small, they shrink toward zero and vanish, which stalls learning in early layers.

Initialization also has to break symmetry. If every weight in a layer starts with the same value, all neurons in that layer compute identical outputs and receive identical gradient updates, so they never learn different features. Random sampling from a carefully scaled distribution solves both problems at once.

The table below shows the typical symptoms of each failure mode.

Initial scaleEffect on signalsObserved symptom
All zeros or constantIdentical neurons, no symmetry breakingLayers behave like a single unit
Too smallActivations and gradients shrink with depthLoss barely moves, early layers stop learning
Too largeActivations saturate or grow without boundLoss becomes NaN or oscillates wildly
Variance-preservingSignal scale stays roughly constantSmooth, fast early convergence
Effects of the initial weight scale on training

Main weight initialization schemes

Modern schemes set the weight variance from layer dimensions so that the variance of outputs, or of backpropagated gradients, matches that of inputs. Here, fan-in is the number of inputs to a neuron and fan-out is the number of outputs, and both are counted per kernel position in convolutional layers.

MethodWeight varianceDesigned for
Xavier (Glorot)2 / (fan-in + fan-out)Tanh, sigmoid and linear layers
He (Kaiming)2 / fan-inReLU and its variants
LeCun1 / fan-inSELU and self-normalizing networks
OrthogonalOrthogonal matrix with chosen gainRecurrent and very deep networks
Scaled residualBase variance divided by depth factorDeep residual networks and Transformers
Common initialization methods and their variance rules

Each rule can be applied with a normal or uniform distribution of the same variance; a uniform version samples from plus or minus the square root of three times the variance. Biases are usually initialized to zero, with exceptions such as a positive forget-gate bias in LSTM cells, often set to 1, to help early memory retention.

Matching initialization to architecture and activation

The correct scheme depends mainly on the activation function and the network topology. He initialization doubles the Xavier variance because ReLU discards roughly half of its inputs, which halves the signal variance; using Xavier with deep ReLU networks therefore lets activations fade.

Normalization layers and residual connections change the picture. Batch normalization and layer normalization rescale activations during training, which makes networks far more tolerant of imperfect initial scales. Residual networks add many branches together, so practitioners often scale down the last layer of each residual branch, sometimes to zero, so that each block starts close to an identity mapping.

ArchitectureRecommended starting pointReason
Multilayer perceptron with ReLUHe normal, zero biasesCompensates for ReLU discarding negatives
Convolutional neural networkHe normal with fan-in modeKeeps feature map variance stable
Recurrent network or LSTMOrthogonal recurrent weights, forget bias 1Limits gradient growth over time steps
TransformerSmall normal std, scaled residual outputsStabilizes deep stacks with layer normalization
Network with SELULeCun normalRequired for self-normalizing behavior
Practical choices by architecture

Transfer learning offers a different path entirely: starting from pretrained weights replaces random initialization for most layers, and only newly added heads need a fresh scheme.

Very large language models add further refinements. Some training recipes use a fixed small standard deviation of about 0.02 for most matrices, while others scale it with the inverse square root of model width. Parameterization schemes such as maximal update parameterization go further and tie initialization to learning-rate scaling, so hyperparameters tuned on a small model transfer to a much larger one with fewer costly experiments.

Common mistakes and practical diagnostics

Most initialization problems are detectable in the first few hundred training steps by monitoring per-layer activation statistics and gradient norms. A healthy network shows activation standard deviations of roughly similar magnitude across layers, rather than a steady rise or fall with depth.

  • Mismatched scheme and activation, such as Xavier with very deep ReLU stacks, causes gradual signal decay.
  • Overriding framework defaults unknowingly: libraries differ in their default schemes, so explicit initialization improves reproducibility.
  • Initializing embeddings too large in language models can destabilize early attention scores.
  • Ignoring random seeds makes results hard to reproduce, because different draws can change final accuracy noticeably in small models.
  • Relying on initialization alone in very deep models; normalization, residual connections and learning-rate warmup are usually needed as well.

When training diverges early, a practical sequence is to check for NaN values, inspect per-layer gradient norms, reduce the initial scale of the final layers and add a short learning-rate warmup before changing the architecture.

The short version

Before a neural network learns anything, all of its internal settings need starting values. Weight initialization is how those first values are chosen.

Picture a long line of people passing a message by whispering. If each person speaks too loudly, the message becomes a shout; if each speaks too softly, it fades away. Good initialization makes every layer pass the signal at the right volume, so learning can start smoothly instead of stalling or blowing up.

Where it makes a difference

A machine learning engineer trains a 50-layer convolutional network for defect detection on factory images. With the library’s Xavier uniform setting and plain ReLU layers without normalization, the training loss stays almost flat for the first two epochs, and per-layer statistics show activation standard deviations dropping by several orders of magnitude between the first and last layers.

Switching to He normal initialization in fan-in mode, with zero biases, keeps activation scales within a narrow band across all layers. The loss starts falling in the first epoch, and the model reaches its target validation accuracy in about half the number of epochs. The engineer documents the scheme and random seed in the training configuration so later experiments remain comparable.