RMSNorm

Also known as: Root mean square normalization

On this page 7

RMSNorm, short for root mean square normalization, is a normalization layer that divides each activation vector by its root mean square (RMS) and multiplies the result by a learned per-feature gain, without subtracting the mean. In neural networks, and above all in large transformer language models, it stabilizes training while costing less computation than layer normalization.

RMSNorm was proposed in 2019 as a simplification of layer normalization, based on the observation that re-scaling invariance, not re-centering, provides most of the benefit. It has since become the default normalization in many open and proprietary large language models, including the LLaMA, Mistral, Gemma and Qwen families.

How RMSNorm rescales activations

RMSNorm operates on one token or example at a time across its feature dimension. For an input vector x with d features, it computes RMS(x) as the square root of the mean of the squared elements plus a small constant epsilon, divides every element by that value, and multiplies element-wise by a learned gain vector g, often written as gamma.

In compact form, the output is y = x / sqrt(mean(x squared) + epsilon) multiplied by g. The result has an RMS of approximately one before the gain is applied, so the scale of activations stays controlled no matter how large or small the incoming values are. Unlike layer normalization, the mean is not subtracted, and most implementations also omit the additive bias term.

The table below lists each element of the computation and its purpose.

ElementDefinitionPurpose
Input vector xActivations of one token, size dValues to normalize
Mean of squaresSum of x squared divided by dMeasures activation magnitude
EpsilonSmall constant, typically 1e-5 or 1e-6Prevents division by zero
RMS(x)Square root of mean of squares plus epsilonScale factor
Gain gLearned vector of size d, initialized to onesRestores per-feature scale
Output yx divided by RMS(x), times gNormalized activations
Elements of the RMSNorm computation

RMSNorm compared with other normalization methods

RMSNorm is best understood as layer normalization without mean centering. Both normalize across features within a single example, which makes them independent of batch size, whereas batch normalization normalizes each feature across the examples in a batch. The comparison below summarizes the practical differences.

MethodStatistics usedNormalization axisLearned parametersTypical setting
RMSNormRoot mean square onlyFeatures of one tokenGainLarge language models
Layer normalizationMean and varianceFeatures of one tokenGain and biasOriginal transformers, BERT, GPT-2
Batch normalizationMean and varianceBatch per featureGain and bias, running averagesConvolutional vision networks
Group normalizationMean and varianceChannel groups per exampleGain and biasVision with small batches
Normalization methods in deep networks

Removing the mean has a specific consequence: RMSNorm is invariant to rescaling of its input but not invariant to shifting it. Empirical work across machine translation, language modeling and vision tasks found that this loss of re-centering rarely hurts quality, while the saved reduction and subtraction steps make the layer measurably faster.

Why large transformers favor RMSNorm

Large transformers favor RMSNorm because it delivers comparable training stability at lower cost, and normalization layers run dozens of times in every forward pass. A model with 32 transformer blocks typically applies normalization at least twice per block, plus a final normalization before the output projection.

PropertyEffect of RMSNormWhy it matters
Compute per callFewer operations than layer normalizationReduces latency in memory-bound kernels
ParametersGain only, no biasSlightly smaller model, simpler fusion
Batch dependenceNoneSame behavior in training and inference
Mixed precisionSensitive to overflow in sum of squaresStatistics are usually computed in float32
Training stabilitySimilar to layer normalizationSupports deep pre-norm stacks
Practical effects of choosing RMSNorm

In practice, the speedup reported in the original evaluation ranged from roughly 7 to 64 percent of normalization time depending on model and implementation, and the end-to-end gain in a full model is smaller. Modern fused GPU kernels narrow the difference further, so the choice today also reflects simplicity and convention rather than speed alone.

Placement of RMSNorm in transformer blocks

RMSNorm is most often used in the pre-norm arrangement, where it normalizes the input of each sublayer before attention or the feedforward network, and the residual connection adds the unnormalized input back afterward. This keeps a clean identity path through the network and helps gradients flow through very deep stacks.

  • Pre-norm: x plus Attention(RMSNorm(x)); the standard choice in current large language models.
  • Post-norm: RMSNorm(x plus Attention(x)); used in the original transformer design with layer normalization, harder to train at depth without learning-rate warmup.
  • Final norm: one RMSNorm after the last block, before the language modeling head.
  • QK-norm: some architectures apply RMSNorm to query and key vectors inside self-attention to limit attention-logit growth.

Implementation pitfalls and limitations

Most RMSNorm problems come from numerical precision and mismatched details between training and inference code. The table pairs common symptoms with their causes and fixes.

SymptomLikely causeFix
NaN or infinite outputsSum of squares overflows in float16Compute statistics in float32
Slightly different outputs after conversionDifferent epsilon valueMatch epsilon exactly to the checkpoint
Quality drop after loading weightsGain stored as offset from oneCheck whether the code adds 1 to g
Unstable training in deep stacksPost-norm placementSwitch to pre-norm
Common RMSNorm problems

RMSNorm also has conceptual limits. Because it does not remove the mean, a large constant offset in activations survives normalization, and research on massive activations in large models shows that a few feature dimensions can dominate the RMS. Some variants, such as partial RMSNorm that estimates RMS from a fraction of features, trade accuracy for speed and are not widely adopted.

The idea in brief

Inside a neural network, numbers flow from layer to layer and can easily grow too large or shrink too small, which makes learning unstable. RMSNorm is a volume leveler for those numbers. It measures how loud a group of values is overall and turns them to a standard level, then lets the network fine-tune each value with its own learned knob.

The older method, layer normalization, also moved every group so that its average sat at zero. RMSNorm skips that step. It turns out that controlling the volume is what matters most, so skipping the extra step saves work without making the model worse.

A practical scenario

A machine learning team is training a 1.3-billion-parameter decoder-only language model with 24 transformer blocks on a cluster of GPUs using bfloat16 mixed precision. Their first prototype copies an older design with post-norm layer normalization, and training diverges twice within the first few thousand steps despite learning-rate warmup.

The team switches to pre-norm RMSNorm with epsilon set to 1e-6, keeps the normalization statistics in float32, and adds a final RMSNorm before the output layer. They also use a fused RMSNorm kernel that combines the normalization with the following matrix multiplication.

The revised model trains without divergence through the full run, and profiling shows a modest reduction in time per step. When the team later exports the model for inference, they document the epsilon value and gain format so that the serving code reproduces the training outputs exactly.