ZeRO

Also known as: Zero Redundancy Optimizer

On this page 7

ZeRO, short for Zero Redundancy Optimizer, is a memory optimization technique for distributed training that partitions model states across data-parallel workers instead of replicating them on every GPU. In AI infrastructure it is the method that lets data parallelism scale to models with tens or hundreds of billions of parameters without rewriting the model for tensor or pipeline parallelism.

ZeRO was introduced by Microsoft researchers and implemented in the DeepSpeed library. Its central observation is that classic data parallelism wastes memory: every GPU holds a full copy of the parameters, gradients and optimizer states, although each worker only needs a slice of them at any given moment.

Where ZeRO finds redundant memory

ZeRO targets the model states, the memory that grows with parameter count. In mixed-precision training with Adam, each parameter typically costs about 16 bytes: 2 bytes for the half-precision weight, 2 bytes for its gradient, and 12 bytes of optimizer state for the full-precision master weight, the momentum and the variance.

The table below breaks down that per-parameter cost, which explains why optimizer states dominate the memory budget.

Model statePrecisionBytes per parameter
ParametersFP16 or BF162
GradientsFP16 or BF162
Master weightsFP324
Adam momentumFP324
Adam varianceFP324
TotalMixed16
Memory per parameter in mixed-precision Adam training

A 7.5-billion-parameter model therefore needs roughly 120 GB of model states before activations are counted, which exceeds the memory of a single 80 GB accelerator. Plain data parallelism duplicates those 120 GB on every GPU; ZeRO removes the duplication.

The three ZeRO stages

ZeRO is applied in three cumulative stages, each partitioning one more category of model state across the data-parallel group. With N workers, every worker stores and updates only 1/N of the partitioned state and receives the rest through collective communication when needed.

The following table shows what each stage partitions and the resulting per-GPU memory for the 7.5-billion-parameter example on 64 GPUs.

StageWhat is partitionedMemory per GPUCommunication volume
Baseline data parallelismNothingAbout 120 GBSame as all-reduce
ZeRO stage 1Optimizer statesAbout 31 GBSame as all-reduce
ZeRO stage 2Optimizer states and gradientsAbout 17 GBSame as all-reduce
ZeRO stage 3Optimizer states, gradients and parametersAbout 2 GBAbout 1.5 times all-reduce
ZeRO stages and their effect on per-GPU memory

Stages 1 and 2 are almost free in communication terms: the usual all-reduce is replaced by a reduce-scatter of gradients followed by an all-gather of updated parameters, which moves the same volume. Stage 3 adds an all-gather of parameters before each layer’s forward and backward pass, raising traffic by about 50 percent in exchange for memory that falls linearly with the number of GPUs.

ZeRO extensions for offloading and faster communication

Later work extended ZeRO beyond GPU memory and reduced its communication cost. These extensions trade bandwidth or latency for capacity, so they suit different hardware situations.

ExtensionCore ideaBest suited for
ZeRO-OffloadKeeps optimizer states and the update step on CPU memorySingle nodes with limited GPU memory
ZeRO-InfinityExtends offloading to CPU and NVMe storageVery large models on modest clusters
ZeRO++Quantizes communication and keeps a secondary parameter copy per nodeClusters with slow inter-node links
ZeRO-RPartitions activations and manages memory fragmentationLong sequences with large activations
Main extensions of the ZeRO family

Offloading works because the Adam update is memory-bound rather than compute-bound, so a CPU can perform it with acceptable speed. The cost is PCIe transfer time, which becomes the bottleneck when batch sizes are small.

ZeRO compared with other parallelism strategies

ZeRO is a form of sharded data parallelism: every worker still processes a different slice of the batch and runs the full model computation, unlike tensor or pipeline parallelism, which split the computation itself. This keeps the programming model simple, because no layer needs to be rewritten.

MethodWhat is splitModel code changesMain cost
ZeRO stage 3Model states across data-parallel workersNone or minimalParameter all-gathers every step
FSDPSame as ZeRO stage 3, native to PyTorchWrapping policy onlyParameter all-gathers every step
Tensor parallelismIndividual matrix operationsSubstantialFrequent intra-layer communication
Pipeline parallelismGroups of layersModeratePipeline bubbles and scheduling
How ZeRO relates to neighboring parallelism methods

In practice these methods are combined rather than chosen exclusively. Large language model training commonly uses tensor parallelism inside a node, pipeline parallelism across a few nodes and ZeRO stage 1 across the remaining data-parallel dimension, an arrangement often called 3D parallelism.

Choosing a stage and common pitfalls

The right ZeRO stage is the lowest stage that fits the model in memory, because each higher stage adds communication or latency. Stage 2 is a frequent default for models up to a few billion parameters; stage 3 becomes necessary when the parameters alone no longer fit on one device.

  • Activations are not covered by stages 1 to 3; activation checkpointing is usually needed alongside ZeRO.
  • Slow interconnects make stage 3 expensive, so InfiniBand or NVLink-class bandwidth strongly affects throughput.
  • Small per-GPU batch sizes reduce the overlap between communication and computation and lower utilization.
  • Checkpoints are sharded by default and must be consolidated before inference or conversion to other frameworks.
  • Memory fragmentation can cause out-of-memory errors even when the theoretical budget fits.

The idea in brief

Training a large AI model on many graphics cards normally means each card keeps a full copy of everything. That wastes a large share of memory. ZeRO shares the bookkeeping instead of copying it.

Imagine a team of eight cooks who each keep the entire recipe book at their station. ZeRO gives each cook only one eighth of the book and lets them pass pages around when needed. The kitchen can then work on a much bigger menu with the same counters.

A practical scenario

A research team wants to fine-tune a 13-billion-parameter language model on two servers with eight 80 GB GPUs each. With plain data parallelism the model states alone would need about 208 GB per GPU, so training fails immediately with out-of-memory errors.

The team enables ZeRO stage 2 first, which brings model states to roughly 11 GB per GPU plus the 26 GB of half-precision weights that every GPU still holds. Combined with activation checkpointing, training fits comfortably and keeps throughput close to the baseline because communication volume does not change. When the team later moves to a 70-billion-parameter model, it switches to ZeRO stage 3 and accepts about 50 percent more network traffic, which the InfiniBand links between the servers absorb with a modest drop in throughput.