Layer normalization, often written LayerNorm, is a technique in neural networks that normalizes the activations of each individual example across its feature dimension, subtracting their mean and dividing by their standard deviation before applying a learnable scale and shift. Introduced by Jimmy Ba, Jamie Ryan Kiros and Geoffrey Hinton in 2016, it was designed to bring the benefits of normalization to recurrent networks, where batch statistics are awkward to compute.
Because its statistics do not depend on the batch, layer normalization behaves identically during training and inference and works with any batch size, including a batch of one. These properties made it the standard normalization layer in the Transformer architecture and in nearly every large language model built on it.
How layer normalization computes its output
Layer normalization transforms a feature vector x of length H in three steps. It first computes the mean of the H values, then their variance, and finally rescales each value so the vector has approximately zero mean and unit variance. Two learned vectors, the gain gamma and the bias beta, then restore whatever scale and offset the network needs.
In compact form the operation is y = gamma * (x - mean) / sqrt(var + eps) + beta. The small constant epsilon prevents division by zero when the variance is tiny. In a transformer, the statistics are computed separately for every token position, over the hidden dimension only, so each token is normalized independently of the others in the sequence and in the batch.
The table below lists the components of the computation and their typical values.
| Component | Role | Typical value or shape |
|---|---|---|
| Mean | Centers the features of one example | Scalar per token |
| Variance | Measures spread across features | Scalar per token, biased estimate |
| Epsilon | Numerical stability term | 1e-5 to 1e-6 |
| Gain gamma | Learned per-feature scale | Vector of length H, initialized to 1 |
| Bias beta | Learned per-feature shift | Vector of length H, initialized to 0 |
The layer therefore adds only 2H trainable parameters, which is negligible compared with the attention and feedforward blocks around it. Its cost lies mainly in memory bandwidth, because it reads and writes the full activation tensor, which is why optimized implementations often fuse it with neighboring operations.
Why normalization stabilizes deep network training
Normalization keeps activations within a predictable numerical range as signals pass through many layers. Without it, small changes in early weights can amplify or shrink activations layer by layer, which produces exploding or vanishing gradients and forces very small learning rates.
Layer normalization also makes each layer’s output invariant to rescaling of its input vector. If an upstream layer multiplies all features by a constant, the normalized result does not change. This invariance smooths the optimization landscape, allowing higher learning rates and faster convergence. The original motivation, reducing internal covariate shift, is still cited, although later research suggests that improved gradient conditioning explains most of the benefit.
Layer normalization compared with other normalization methods
Normalization methods differ mainly in which axes they average over. The choice determines whether the layer depends on batch size, how it treats spatial structure and which architectures it suits.
| Method | Statistics computed over | Batch dependent | Typical use |
|---|---|---|---|
| Layer normalization | All features of one example | No | Transformers, RNNs |
| Batch normalization | One feature across the batch | Yes | Convolutional networks |
| Instance normalization | One channel of one image | No | Style transfer |
| Group normalization | Groups of channels of one image | No | Detection with small batches |
| RMSNorm | All features, root mean square only | No | Many recent language models |
The contrast with batch normalization is the most important one. Batch normalization needs running averages for inference and degrades when batches are small or sequences vary in length, whereas layer normalization has no such state. RMSNorm simplifies layer normalization further by skipping mean subtraction and the bias term, saving computation with little or no loss in quality.
Placement in transformer architectures
Where layer normalization sits relative to the residual connection strongly affects training stability. The original 2017 Transformer used Post-LN, normalizing after the residual addition, while GPT-2 and most later models adopted Pre-LN, normalizing the input to each sublayer before attention or the feedforward block.
| Placement | Position | Strength | Weakness |
|---|---|---|---|
| Post-LN | After residual addition | Strong final quality when it trains | Needs learning-rate warmup; unstable when deep |
| Pre-LN | Before each sublayer | Stable gradients in very deep stacks | Residual stream grows; may need a final LayerNorm |
| Sandwich or hybrid | Before and after sublayer | Controls activation growth | Extra computation and tuning |
| QK normalization | On attention queries and keys | Prevents attention logit blow-up | Adds layers inside attention |
With Pre-LN, the residual path remains an unnormalized identity route, so gradients flow directly to early layers. This is why Pre-LN models typically add one final normalization layer before the output projection to control the magnitude of the accumulated residual stream.
Implementation pitfalls and common mistakes
Layer normalization is simple to describe yet easy to implement subtly wrong. Most problems involve numerical precision, the normalized axis or the way its parameters are optimized.
| Symptom | Likely cause | Fix |
|---|---|---|
| NaN loss in mixed precision | Variance computed in 16-bit floats | Compute statistics in FP32 |
| Model ignores sequence order cues | Normalized over sequence axis by mistake | Normalize over the hidden dimension only |
| Loss spikes in deep Post-LN models | Unstable gradients near output layers | Switch to Pre-LN or add warmup |
| Slow convergence after fine-tuning | Weight decay applied to gamma and beta | Exclude normalization parameters from decay |
A further subtlety concerns epsilon placement. Some libraries add epsilon inside the square root and others add it to the standard deviation, so porting weights between frameworks can produce small but measurable output differences unless the convention is matched.
A plain-language summary
Think of a classroom where each student’s test answers are graded on very different scales. Before comparing anything, a teacher converts each student’s set of scores to the same scale. Layer normalization does this for the numbers inside a neural network.
For every single input, it takes all the values a layer produces, centers them and evens out their spread. The network then learns how much to stretch or shift them back. The result is that training stays steady and predictable, even in very deep models, and it works the same way whether the network sees one example or thousands.
An example from practice
A machine learning team trains a 24-layer transformer for customer support text classification on GPUs with limited memory, which restricts them to a batch size of 8 sequences. An early prototype using batch normalization produced noisy statistics and unstable validation accuracy, because eight variable-length sequences gave unreliable batch averages.
The team switches to Pre-LN layer normalization with epsilon of 1e-5, computes the statistics in FP32 under mixed-precision training, and excludes gamma and beta from weight decay. Training loss now falls smoothly, the learning rate can be raised by about a factor of three, and the model reaches its target accuracy in roughly 40 percent fewer training steps. Inference on single customer messages produces exactly the same normalization behavior as training.