Mixed-precision training

On this page 8

Mixed-precision training is a technique that performs most of the arithmetic of neural network training in low-precision number formats, such as FP16, BF16 or FP8, while keeping numerically sensitive values, above all the master copy of the weights, in 32-bit floating point. In artificial intelligence it is the default way to train large models, because it roughly halves memory traffic and unlocks the specialized matrix units of modern accelerators without measurably hurting final model quality.

The idea is simple to state and subtle to execute: use cheap precision where errors average out, and full precision where errors accumulate. Within Training, it sits next to gradient checkpointing and gradient accumulation as one of the core efficiency techniques that make pretraining at scale affordable.

How mixed-precision training works

Mixed-precision training works by assigning each tensor in the training loop the cheapest format that still keeps it numerically safe. Forward activations, backward gradients and matrix multiplications run in 16-bit or 8-bit formats, while weight updates, optimizer states and large reductions stay in FP32.

A typical step follows a fixed pattern. The optimizer holds FP32 master weights; before the forward pass, a low-precision copy is cast for computation. Matrix multiplications multiply 16-bit inputs but accumulate partial sums in FP32 inside the hardware, which prevents rounding errors from compounding across thousands of additions. Gradients are produced in low precision, converted back, and applied to the FP32 master weights by the optimizer.

The master copy matters because individual updates are tiny. With a learning rate of 1e-4, an update may be thousands of times smaller than the weight it modifies. FP16 has only about 11 bits of significand, so adding such an update to a 16-bit weight often rounds to zero and the model silently stops learning. Keeping weights in FP32 preserves these small contributions.

The following table shows where each tensor usually lives in a standard BF16 recipe.

Tensor or operationUsual formatReason
Master weightsFP32Small updates must not round away
Optimizer states (Adam moments)FP32Second moments span a wide range
Matrix multiplication inputsBF16 or FP16Uses tensor cores at full speed
Matrix multiplication accumulationFP32Limits rounding error across long sums
Softmax, layer norm, lossFP32Exponentials and reductions overflow easily
Activations saved for backwardBF16 or FP16Halves activation memory
Typical precision assignment in a BF16 mixed-precision recipe

Number formats used in mixed precision

Floating-point formats differ in how they split bits between exponent (range) and significand (precision), and that split decides which failure mode a format is prone to. The IEEE 754 standard defines FP32 and FP16; BF16 and the FP8 variants were introduced for machine learning hardware.

FormatExponent and significand bitsApproximate rangeMain risk
FP328 and 23About 1e-38 to 3e38Slow and memory-heavy
FP165 and 10About 6e-8 to 65,504Overflow and gradient underflow
BF168 and 7Same as FP32Coarse rounding of small differences
FP8 E4M34 and 3Up to about 448Needs per-tensor scaling
FP8 E5M25 and 2Up to about 57,344Very low precision, used for gradients
Floating-point formats compared

BF16 keeps the FP32 exponent range, so it rarely overflows or underflows and usually needs no special handling. FP16 has more precision but a narrow range, which is why it requires loss scaling. FP8 training, supported on recent accelerators, needs per-tensor or per-block scaling factors that are updated continuously, and it is still treated as an advanced recipe rather than a default.

Loss scaling and numerical stability

Loss scaling is the technique that keeps small FP16 gradients from underflowing to zero. Many gradient values in deep networks are smaller than the smallest positive FP16 number, so they vanish during the backward pass unless they are shifted into the representable range.

The fix multiplies the loss by a scale factor such as 2^16 before backpropagation, which shifts every gradient upward by the same amount through the chain rule. Before the optimizer step, gradients are divided by the same factor in FP32. Dynamic loss scaling automates the choice: the scaler doubles the factor after a run of stable steps and halves it, skipping that step, whenever an overflow produces infinities or NaN values.

Stability problems have recognizable signatures. The table below maps common symptoms to their usual causes and remedies.

SymptomLikely causeTypical fix
Loss becomes NaN earlyFP16 overflow in attention or softmaxRun softmax in FP32 or switch to BF16
Loss plateaus unexpectedlyGradients underflow to zeroEnable or raise loss scaling
Frequent skipped stepsScale factor too aggressiveLower the initial scale or growth rate
Slow drift versus FP32 baselineWeights stored in low precisionKeep FP32 master weights
Sudden loss spikesOutlier activations exceed rangeAdd gradient clipping and check normalization
Mixed-precision symptoms and fixes

Benefits for speed and memory

The main benefit of mixed-precision training is higher throughput at the same model quality. Tensor cores and similar matrix engines deliver several times more operations per second in 16-bit than in FP32, and FP8 roughly doubles that again on hardware that supports it.

Memory savings are real but smaller than many expect. Activations and low-precision weight copies shrink by half, which allows a larger batch size or longer sequences. However, FP32 master weights and Adam states remain, so a model trained with Adam still needs roughly 16 bytes per parameter for weights, gradients and optimizer states, before activations. Sharded optimizers and offloading address that remaining cost separately.

  • Faster matrix math, typically a 2x to 3x step-time improvement over pure FP32 for transformer workloads.
  • Lower activation memory, often the dominant cost for long sequences.
  • Reduced communication volume when gradients are exchanged in 16-bit between devices.
  • No change to the model architecture or the training objective.

Mixed-precision training is often confused with quantization, but the two serve different stages of the model lifecycle. The comparison below separates it from neighboring methods in the same toolbox.

TechniqueMain goalWhen it appliesEffect on weights
Mixed-precision trainingFaster, cheaper trainingDuring trainingFP32 master copy retained
QuantizationSmaller, faster inferenceAfter or near the end of trainingStored in INT8, INT4 or FP8
Gradient checkpointingLower activation memoryDuring trainingUnchanged
Gradient accumulationLarger effective batch sizeDuring trainingUnchanged
QLoRAMemory-efficient fine-tuningFine-tuningFrozen 4-bit base plus 16-bit adapters
Related efficiency techniques

These methods are complementary rather than competing. A large pretraining run commonly combines BF16 mixed precision, gradient checkpointing and sharded optimizer states, while QLoRA itself relies on 16-bit computation over a quantized base model.

Limitations and common mistakes

Mixed-precision training is robust in mainstream frameworks, yet several failure modes recur in practice, especially when custom layers or new number formats are involved.

  • Casting custom operations blindly: user-written losses or normalization layers may need explicit FP32 execution.
  • Comparing runs across formats: FP16, BF16 and FP8 runs are not bit-identical, so small metric differences are expected.
  • Assuming FP8 is a drop-in switch: it requires careful scaling recipes and validation against a BF16 baseline.
  • Ignoring hardware support: older GPUs lack fast BF16, and emulated formats can be slower than FP32.
  • Saving checkpoints only in 16-bit: resuming without FP32 master weights and optimizer states degrades training.

The simple explanation

Training an AI model means doing an enormous number of multiplications. Mixed-precision training does most of them with shorter, rougher numbers, which computers process much faster, and keeps a precise version for the parts that matter.

Think of an accountant who estimates daily receipts in round figures but records the official ledger to the cent. The quick estimates save time, and the careful ledger makes sure that small amounts are never lost. The result is nearly the same model in far less time.

How it is used in practice

A research team plans to pretrain a 7-billion-parameter language model on a cluster of 64 modern GPUs. In a short FP32 test, each training step takes so long that the full run would exceed the compute budget by months, and activation memory forces a small batch size.

The team switches to BF16 mixed precision with FP32 master weights, keeps softmax and layer normalization in FP32, and adds gradient clipping at 1.0. Because BF16 shares the FP32 exponent range, no loss scaling is needed. Step time drops to roughly one third, and the freed activation memory allows a larger micro-batch.

Before committing, the team trains a small proxy model in both FP32 and BF16 for a few thousand steps. The loss curves overlap within normal run-to-run noise, so the full run proceeds in BF16 and finishes within the planned budget.