Mixed-precision training is a technique that performs most of the arithmetic of neural network training in low-precision number formats, such as FP16, BF16 or FP8, while keeping numerically sensitive values, above all the master copy of the weights, in 32-bit floating point. In artificial intelligence it is the default way to train large models, because it roughly halves memory traffic and unlocks the specialized matrix units of modern accelerators without measurably hurting final model quality.
The idea is simple to state and subtle to execute: use cheap precision where errors average out, and full precision where errors accumulate. Within Training, it sits next to gradient checkpointing and gradient accumulation as one of the core efficiency techniques that make pretraining at scale affordable.
How mixed-precision training works
Mixed-precision training works by assigning each tensor in the training loop the cheapest format that still keeps it numerically safe. Forward activations, backward gradients and matrix multiplications run in 16-bit or 8-bit formats, while weight updates, optimizer states and large reductions stay in FP32.
A typical step follows a fixed pattern. The optimizer holds FP32 master weights; before the forward pass, a low-precision copy is cast for computation. Matrix multiplications multiply 16-bit inputs but accumulate partial sums in FP32 inside the hardware, which prevents rounding errors from compounding across thousands of additions. Gradients are produced in low precision, converted back, and applied to the FP32 master weights by the optimizer.
The master copy matters because individual updates are tiny. With a learning rate of 1e-4, an update may be thousands of times smaller than the weight it modifies. FP16 has only about 11 bits of significand, so adding such an update to a 16-bit weight often rounds to zero and the model silently stops learning. Keeping weights in FP32 preserves these small contributions.
The following table shows where each tensor usually lives in a standard BF16 recipe.
| Tensor or operation | Usual format | Reason |
|---|---|---|
| Master weights | FP32 | Small updates must not round away |
| Optimizer states (Adam moments) | FP32 | Second moments span a wide range |
| Matrix multiplication inputs | BF16 or FP16 | Uses tensor cores at full speed |
| Matrix multiplication accumulation | FP32 | Limits rounding error across long sums |
| Softmax, layer norm, loss | FP32 | Exponentials and reductions overflow easily |
| Activations saved for backward | BF16 or FP16 | Halves activation memory |
Number formats used in mixed precision
Floating-point formats differ in how they split bits between exponent (range) and significand (precision), and that split decides which failure mode a format is prone to. The IEEE 754 standard defines FP32 and FP16; BF16 and the FP8 variants were introduced for machine learning hardware.
| Format | Exponent and significand bits | Approximate range | Main risk |
|---|---|---|---|
| FP32 | 8 and 23 | About 1e-38 to 3e38 | Slow and memory-heavy |
| FP16 | 5 and 10 | About 6e-8 to 65,504 | Overflow and gradient underflow |
| BF16 | 8 and 7 | Same as FP32 | Coarse rounding of small differences |
| FP8 E4M3 | 4 and 3 | Up to about 448 | Needs per-tensor scaling |
| FP8 E5M2 | 5 and 2 | Up to about 57,344 | Very low precision, used for gradients |
BF16 keeps the FP32 exponent range, so it rarely overflows or underflows and usually needs no special handling. FP16 has more precision but a narrow range, which is why it requires loss scaling. FP8 training, supported on recent accelerators, needs per-tensor or per-block scaling factors that are updated continuously, and it is still treated as an advanced recipe rather than a default.
Loss scaling and numerical stability
Loss scaling is the technique that keeps small FP16 gradients from underflowing to zero. Many gradient values in deep networks are smaller than the smallest positive FP16 number, so they vanish during the backward pass unless they are shifted into the representable range.
The fix multiplies the loss by a scale factor such as 2^16 before backpropagation, which shifts every gradient upward by the same amount through the chain rule. Before the optimizer step, gradients are divided by the same factor in FP32. Dynamic loss scaling automates the choice: the scaler doubles the factor after a run of stable steps and halves it, skipping that step, whenever an overflow produces infinities or NaN values.
Stability problems have recognizable signatures. The table below maps common symptoms to their usual causes and remedies.
| Symptom | Likely cause | Typical fix |
|---|---|---|
| Loss becomes NaN early | FP16 overflow in attention or softmax | Run softmax in FP32 or switch to BF16 |
| Loss plateaus unexpectedly | Gradients underflow to zero | Enable or raise loss scaling |
| Frequent skipped steps | Scale factor too aggressive | Lower the initial scale or growth rate |
| Slow drift versus FP32 baseline | Weights stored in low precision | Keep FP32 master weights |
| Sudden loss spikes | Outlier activations exceed range | Add gradient clipping and check normalization |
Benefits for speed and memory
The main benefit of mixed-precision training is higher throughput at the same model quality. Tensor cores and similar matrix engines deliver several times more operations per second in 16-bit than in FP32, and FP8 roughly doubles that again on hardware that supports it.
Memory savings are real but smaller than many expect. Activations and low-precision weight copies shrink by half, which allows a larger batch size or longer sequences. However, FP32 master weights and Adam states remain, so a model trained with Adam still needs roughly 16 bytes per parameter for weights, gradients and optimizer states, before activations. Sharded optimizers and offloading address that remaining cost separately.
- Faster matrix math, typically a 2x to 3x step-time improvement over pure FP32 for transformer workloads.
- Lower activation memory, often the dominant cost for long sequences.
- Reduced communication volume when gradients are exchanged in 16-bit between devices.
- No change to the model architecture or the training objective.
Mixed precision compared with related techniques
Mixed-precision training is often confused with quantization, but the two serve different stages of the model lifecycle. The comparison below separates it from neighboring methods in the same toolbox.
| Technique | Main goal | When it applies | Effect on weights |
|---|---|---|---|
| Mixed-precision training | Faster, cheaper training | During training | FP32 master copy retained |
| Quantization | Smaller, faster inference | After or near the end of training | Stored in INT8, INT4 or FP8 |
| Gradient checkpointing | Lower activation memory | During training | Unchanged |
| Gradient accumulation | Larger effective batch size | During training | Unchanged |
| QLoRA | Memory-efficient fine-tuning | Fine-tuning | Frozen 4-bit base plus 16-bit adapters |
These methods are complementary rather than competing. A large pretraining run commonly combines BF16 mixed precision, gradient checkpointing and sharded optimizer states, while QLoRA itself relies on 16-bit computation over a quantized base model.
Limitations and common mistakes
Mixed-precision training is robust in mainstream frameworks, yet several failure modes recur in practice, especially when custom layers or new number formats are involved.
- Casting custom operations blindly: user-written losses or normalization layers may need explicit FP32 execution.
- Comparing runs across formats: FP16, BF16 and FP8 runs are not bit-identical, so small metric differences are expected.
- Assuming FP8 is a drop-in switch: it requires careful scaling recipes and validation against a BF16 baseline.
- Ignoring hardware support: older GPUs lack fast BF16, and emulated formats can be slower than FP32.
- Saving checkpoints only in 16-bit: resuming without FP32 master weights and optimizer states degrades training.
The simple explanation
Training an AI model means doing an enormous number of multiplications. Mixed-precision training does most of them with shorter, rougher numbers, which computers process much faster, and keeps a precise version for the parts that matter.
Think of an accountant who estimates daily receipts in round figures but records the official ledger to the cent. The quick estimates save time, and the careful ledger makes sure that small amounts are never lost. The result is nearly the same model in far less time.
How it is used in practice
A research team plans to pretrain a 7-billion-parameter language model on a cluster of 64 modern GPUs. In a short FP32 test, each training step takes so long that the full run would exceed the compute budget by months, and activation memory forces a small batch size.
The team switches to BF16 mixed precision with FP32 master weights, keeps softmax and layer normalization in FP32, and adds gradient clipping at 1.0. Because BF16 shares the FP32 exponent range, no loss scaling is needed. Step time drops to roughly one third, and the freed activation memory allows a larger micro-batch.
Before committing, the team trains a small proxy model in both FP32 and BF16 for a few thousand steps. The loss curves overlap within normal run-to-run noise, so the full run proceeds in BF16 and finishes within the planned budget.