Layer normalization

Also known as: LayerNorm

On this page 7

Layer normalization, often written LayerNorm, is a technique in neural networks that normalizes the activations of each individual example across its feature dimension, subtracting their mean and dividing by their standard deviation before applying a learnable scale and shift. Introduced by Jimmy Ba, Jamie Ryan Kiros and Geoffrey Hinton in 2016, it was designed to bring the benefits of normalization to recurrent networks, where batch statistics are awkward to compute.

Because its statistics do not depend on the batch, layer normalization behaves identically during training and inference and works with any batch size, including a batch of one. These properties made it the standard normalization layer in the Transformer architecture and in nearly every large language model built on it.

How layer normalization computes its output

Layer normalization transforms a feature vector x of length H in three steps. It first computes the mean of the H values, then their variance, and finally rescales each value so the vector has approximately zero mean and unit variance. Two learned vectors, the gain gamma and the bias beta, then restore whatever scale and offset the network needs.

In compact form the operation is y = gamma * (x - mean) / sqrt(var + eps) + beta. The small constant epsilon prevents division by zero when the variance is tiny. In a transformer, the statistics are computed separately for every token position, over the hidden dimension only, so each token is normalized independently of the others in the sequence and in the batch.

The table below lists the components of the computation and their typical values.

ComponentRoleTypical value or shape
MeanCenters the features of one exampleScalar per token
VarianceMeasures spread across featuresScalar per token, biased estimate
EpsilonNumerical stability term1e-5 to 1e-6
Gain gammaLearned per-feature scaleVector of length H, initialized to 1
Bias betaLearned per-feature shiftVector of length H, initialized to 0
Components of the layer normalization computation

The layer therefore adds only 2H trainable parameters, which is negligible compared with the attention and feedforward blocks around it. Its cost lies mainly in memory bandwidth, because it reads and writes the full activation tensor, which is why optimized implementations often fuse it with neighboring operations.

Why normalization stabilizes deep network training

Normalization keeps activations within a predictable numerical range as signals pass through many layers. Without it, small changes in early weights can amplify or shrink activations layer by layer, which produces exploding or vanishing gradients and forces very small learning rates.

Layer normalization also makes each layer’s output invariant to rescaling of its input vector. If an upstream layer multiplies all features by a constant, the normalized result does not change. This invariance smooths the optimization landscape, allowing higher learning rates and faster convergence. The original motivation, reducing internal covariate shift, is still cited, although later research suggests that improved gradient conditioning explains most of the benefit.

Layer normalization compared with other normalization methods

Normalization methods differ mainly in which axes they average over. The choice determines whether the layer depends on batch size, how it treats spatial structure and which architectures it suits.

MethodStatistics computed overBatch dependentTypical use
Layer normalizationAll features of one exampleNoTransformers, RNNs
Batch normalizationOne feature across the batchYesConvolutional networks
Instance normalizationOne channel of one imageNoStyle transfer
Group normalizationGroups of channels of one imageNoDetection with small batches
RMSNormAll features, root mean square onlyNoMany recent language models
Normalization methods and the axes they use

The contrast with batch normalization is the most important one. Batch normalization needs running averages for inference and degrades when batches are small or sequences vary in length, whereas layer normalization has no such state. RMSNorm simplifies layer normalization further by skipping mean subtraction and the bias term, saving computation with little or no loss in quality.

Placement in transformer architectures

Where layer normalization sits relative to the residual connection strongly affects training stability. The original 2017 Transformer used Post-LN, normalizing after the residual addition, while GPT-2 and most later models adopted Pre-LN, normalizing the input to each sublayer before attention or the feedforward block.

PlacementPositionStrengthWeakness
Post-LNAfter residual additionStrong final quality when it trainsNeeds learning-rate warmup; unstable when deep
Pre-LNBefore each sublayerStable gradients in very deep stacksResidual stream grows; may need a final LayerNorm
Sandwich or hybridBefore and after sublayerControls activation growthExtra computation and tuning
QK normalizationOn attention queries and keysPrevents attention logit blow-upAdds layers inside attention
Common placements of layer normalization in transformer blocks

With Pre-LN, the residual path remains an unnormalized identity route, so gradients flow directly to early layers. This is why Pre-LN models typically add one final normalization layer before the output projection to control the magnitude of the accumulated residual stream.

Implementation pitfalls and common mistakes

Layer normalization is simple to describe yet easy to implement subtly wrong. Most problems involve numerical precision, the normalized axis or the way its parameters are optimized.

SymptomLikely causeFix
NaN loss in mixed precisionVariance computed in 16-bit floatsCompute statistics in FP32
Model ignores sequence order cuesNormalized over sequence axis by mistakeNormalize over the hidden dimension only
Loss spikes in deep Post-LN modelsUnstable gradients near output layersSwitch to Pre-LN or add warmup
Slow convergence after fine-tuningWeight decay applied to gamma and betaExclude normalization parameters from decay
Frequent layer normalization problems and their fixes

A further subtlety concerns epsilon placement. Some libraries add epsilon inside the square root and others add it to the standard deviation, so porting weights between frameworks can produce small but measurable output differences unless the convention is matched.

A plain-language summary

Think of a classroom where each student’s test answers are graded on very different scales. Before comparing anything, a teacher converts each student’s set of scores to the same scale. Layer normalization does this for the numbers inside a neural network.

For every single input, it takes all the values a layer produces, centers them and evens out their spread. The network then learns how much to stretch or shift them back. The result is that training stays steady and predictable, even in very deep models, and it works the same way whether the network sees one example or thousands.

An example from practice

A machine learning team trains a 24-layer transformer for customer support text classification on GPUs with limited memory, which restricts them to a batch size of 8 sequences. An early prototype using batch normalization produced noisy statistics and unstable validation accuracy, because eight variable-length sequences gave unreliable batch averages.

The team switches to Pre-LN layer normalization with epsilon of 1e-5, computes the statistics in FP32 under mixed-precision training, and excludes gamma and beta from weight decay. Training loss now falls smoothly, the learning rate can be raised by about a factor of three, and the model reaches its target accuracy in roughly 40 percent fewer training steps. Inference on single customer messages produces exactly the same normalization behavior as training.