Batch normalization

Also known as: BatchNorm

On this page 7

Batch normalization, often called BatchNorm, is a layer in neural networks that standardizes the activations of each feature using the mean and variance computed over the current mini-batch, then rescales them with two learned parameters. Introduced in 2015, it made very deep networks noticeably easier to train and became a standard building block of convolutional architectures across Artificial intelligence.

By keeping intermediate activations in a predictable range, batch normalization allows higher learning rates, reduces sensitivity to weight initialization and adds a mild regularizing effect. Its reliance on batch statistics, however, also creates behavior that differs between training and inference, which is the source of most practical problems with it.

How batch normalization transforms activations

Batch normalization works by subtracting the mini-batch mean and dividing by the mini-batch standard deviation for every feature, so each feature has roughly zero mean and unit variance within the batch. A small constant, epsilon, is added to the variance to prevent division by zero. The layer then multiplies the result by a learned scale, gamma, and adds a learned shift, beta, so the network can recover any distribution it actually needs.

In a fully connected layer, statistics are computed per neuron across the batch. In a convolutional layer, they are computed per channel across the batch and all spatial positions, which means one gamma and one beta per feature map. The table below lists the quantities involved.

QuantityLearned or computedPurpose
Batch meanComputed per batchCenters each feature around zero
Batch varianceComputed per batchScales each feature to unit variance
EpsilonFixed constant, often 1e-5Keeps the division numerically stable
Gamma (scale)Learned by gradient descentRestores a useful spread of values
Beta (shift)Learned by gradient descentRestores a useful offset
Running mean and varianceExponential moving averageReplace batch statistics at inference
Quantities in a batch normalization layer

Because normalization is part of the computational graph, gradients flow through the mean and variance as well. Backpropagation therefore accounts for the fact that changing one example changes the statistics applied to every other example in the batch.

Why it helps optimization

Batch normalization speeds up training mainly by making the loss landscape smoother and the gradients better conditioned. The original explanation was a reduction of internal covariate shift, meaning the changing distribution of layer inputs as earlier weights update. Later analysis showed that this explanation is incomplete and that the smoothing of the optimization surface accounts for much of the benefit.

In practice, the layer makes networks less sensitive to weight scale, which lets them tolerate learning rates several times higher than an equivalent unnormalized network. It also eases the vanishing gradient problem in deep stacks, although residual connections remain the primary fix for that issue in very deep models.

The noise introduced by estimating statistics from a finite batch acts as a weak regularizer, similar in spirit to dropout. This effect is real but modest, and it shrinks as batch size grows.

Training mode versus inference mode

Batch normalization behaves differently during training and during inference. During training it uses the statistics of the current batch and updates the running averages. During inference it uses the stored running mean and variance, so the output for one example does not depend on which other examples happen to be processed with it.

The following table contrasts the two modes.

AspectTraining modeInference mode
Statistics usedCurrent mini-batchRunning averages from training
Dependence between examplesYes, within the batchNone
Running averagesUpdated with a momentum factorFrozen
DeterminismVaries with batch compositionDeterministic per input
Can be fused into prior layerNoYes, into weights and bias
Behavior of batch normalization by mode

Forgetting to switch a model into evaluation mode is one of the most common deployment bugs. Predictions then change with batch size and composition, and accuracy on single-example requests can drop sharply. At inference, the normalization reduces to a fixed affine transform, so it can be folded into the preceding convolution at no runtime cost.

Batch normalization compared with other normalization layers

Batch normalization is one of several normalization schemes that differ in which axes the statistics are computed over. The choice depends on architecture, batch size and whether examples must be processed independently.

The table below compares batch normalization with its closest alternatives.

MethodStatistics computed overBatch dependenceTypical use
Batch normalizationBatch and spatial positions, per channelYesConvolutional networks with large batches
Layer normalizationAll features of one exampleNoTransformers and recurrent networks
RMSNormRoot mean square of one exampleNoLarge language models
Group normalizationChannel groups of one exampleNoDetection and segmentation with small batches
Instance normalizationSpatial positions of one channel and exampleNoStyle transfer and image generation
Normalization layers and their typical use

Transformers almost always use layer normalization or RMSNorm instead of batch normalization, because sequence lengths vary, padding distorts batch statistics and autoregressive generation processes one token at a time.

Practical guidance and common mistakes

Batch normalization works reliably when batches are large enough to give stable statistics, typically 32 or more examples per device for image classification. Below roughly 8 examples per device, estimates become noisy and accuracy often degrades, which makes group normalization or synchronized batch normalization across devices a better option.

The table below lists frequent problems, their causes and the usual fix.

SymptomLikely causeFix
Good training accuracy, poor test accuracyModel left in training modeSwitch to evaluation mode before inference
Unstable loss with small batchesNoisy batch statisticsUse group normalization or synchronized statistics
Accuracy drop after fine-tuningRunning averages overwritten by new dataFreeze normalization statistics when data is small
Redundant parametersBias in the layer before normalizationRemove that bias, since beta replaces it
Mismatch between domainsRunning statistics from a different distributionRecompute statistics on target data
Common batch normalization problems

Placement also matters. The original design placed the layer before the activation function, while many modern residual networks place it inside the residual branch, either before or after ReLU. Both orders work, but the choice should stay consistent with the weights being loaded.

In simple terms

A neural network passes numbers from layer to layer, and those numbers can drift to very large or very small values as training changes the weights. Batch normalization rescales the numbers at each step so they stay in a comfortable range, much like a sound engineer keeps every microphone at a similar volume before mixing.

Because each layer receives well-behaved input, the network learns faster and more reliably. After training, the layer simply uses the average levels it remembered, so every input is treated the same way.

A practical example

A computer vision team trains a 50-layer residual network to classify defects in photographs of circuit boards. Without normalization, training diverges whenever the learning rate exceeds 0.01, and reaching acceptable accuracy takes about 90 epochs.

After adding batch normalization after every convolution and using a batch of 64 images per GPU, the team raises the learning rate to 0.1 and reaches the same validation accuracy in roughly a third of the epochs. Before deployment, the engineers fold each normalization layer into its convolution, so the production model runs with no extra latency.

An early test on the factory line shows unstable single-image predictions. The cause is a script that left the model in training mode, and switching to evaluation mode restores the expected accuracy.