DeepSpeed

On this page 7

DeepSpeed is an open-source deep learning optimization library for PyTorch, originally developed by Microsoft, that makes it practical to train and serve models with billions of parameters across many GPUs. Among AI frameworks & tools, it occupies the systems layer: it does not define model architectures, but it decides how memory, computation and communication are distributed when a model is too large or too slow for a single accelerator.

Its best-known contribution is the Zero Redundancy Optimizer (ZeRO), a family of techniques that removes duplicated training state from data-parallel workers. Around ZeRO, the library adds mixed precision training, pipeline parallelism, offloading to CPU memory and NVMe storage, communication compression and an inference engine, all controlled through a single JSON configuration file.

How DeepSpeed reduces training memory with ZeRO

DeepSpeed reduces memory by partitioning training state across data-parallel ranks instead of replicating it on every GPU. In classic data parallelism, each worker holds a full copy of the parameters, gradients and optimizer states, which wastes memory in proportion to the number of workers.

The cost is large because mixed precision Adam needs about 16 bytes per parameter: 2 bytes for half-precision weights, 2 bytes for gradients and roughly 12 bytes for the fp32 master weights, momentum and variance. A 7-billion-parameter model therefore needs around 112 GB of training state before activations are counted, far beyond one 80 GB accelerator. ZeRO splits this state into shards and gathers pieces only when a layer needs them.

The following table shows what each ZeRO stage partitions and what it costs in communication.

StageWhat is partitionedMemory savingCommunication overhead
ZeRO stage 1Optimizer statesUp to about 4xSame as standard data parallelism
ZeRO stage 2Optimizer states and gradientsUp to about 8xSame as standard data parallelism
ZeRO stage 3Optimizer states, gradients and parametersScales linearly with GPU countAbout 1.5x of data parallelism
ZeRO-OffloadOptimizer states and computation moved to CPUFits larger models on few GPUsLimited by PCIe bandwidth
ZeRO-InfinityState spilled to CPU and NVMeTrillion-parameter scale in principleLimited by storage throughput
ZeRO stages and their effect on memory

Stage 3 behaves like fully sharded data parallelism: each layer’s weights are all-gathered just before the forward or backward pass and released afterward. A later refinement, ZeRO++, quantizes the communicated weights and gradients and keeps a secondary partition inside each node to cut cross-node traffic on clusters with slower interconnects.

Main components of the DeepSpeed library

DeepSpeed is a collection of cooperating subsystems rather than one algorithm, and most projects enable only a few of them. The engine wraps a PyTorch model, optimizer and data loader through deepspeed.initialize, and the deepspeed launcher starts one process per GPU across nodes.

The table below summarizes the components that practitioners encounter most often.

ComponentPurposeTypical use
Training engineWraps model, optimizer and schedulerEvery DeepSpeed training run
ZeRO optimizerShards training state across ranksModels that exceed GPU memory
Pipeline engineSplits layers into sequential stagesVery deep models on many nodes
Activation checkpointingRecomputes activations in backward passLong sequences and large batches
DeepSpeed-UlyssesSequence parallelism for attentionContext lengths beyond 100,000 tokens
DeepSpeed-InferenceOptimized kernels and tensor parallelismServing large models with low latency
Core DeepSpeed components

Configuration lives in a JSON file such as `ds_config.json`, which sets the batch size, gradient accumulation steps, precision mode (fp16 or bf16), ZeRO stage, offload targets and optimizer. Because the model code stays unchanged, the same script can move from one GPU to hundreds by changing only the configuration.

DeepSpeed complements rather than replaces PyTorch, and it overlaps with several neighboring tools in the same toolchain. Choosing among them depends on how much control, portability and scale a team needs.

ToolMain roleRelationship to DeepSpeed
PyTorch FSDPNative fully sharded data parallelismFunctional alternative to ZeRO stage 3
Megatron-LMTensor and pipeline parallel transformersOften combined with DeepSpeed for 3D parallelism
Hugging Face AccelerateHardware-agnostic training launcherCan use DeepSpeed as a backend
PyTorch LightningStructured training loopsOffers a DeepSpeed strategy
RayDistributed compute and orchestrationSchedules jobs that may run DeepSpeed inside
How DeepSpeed relates to neighboring tools

In practice, many teams never call DeepSpeed directly: the Hugging Face Transformers Trainer accepts a DeepSpeed configuration file and handles initialization. PyTorch FSDP has narrowed the gap since its introduction, so the decision often rests on offloading needs, existing configurations and familiarity rather than raw capability.

Choosing a DeepSpeed configuration

The right configuration is the lowest ZeRO stage that fits the model, because each higher stage adds communication. Engineers usually start with stage 2, move to stage 3 only when parameters no longer fit, and turn on offloading as a last resort.

SituationRecommended settingReason
Model fits, optimizer does notZeRO stage 1 or 2Saves memory with no extra traffic
Parameters exceed one GPUZeRO stage 3Shards the weights themselves
Few GPUs, large CPU memoryZeRO-OffloadTrades speed for capacity
Fine-tuning on one nodeStage 2 with bf16Simple and numerically stable
Hundreds of GPUs, slow networkZeRO++ or pipeline parallelismReduces cross-node communication
Decision guide for common situations

Precision matters as much as sharding. bf16 is preferred on recent accelerators because its wide exponent range avoids the loss scaling that fp16 requires; fp16 remains necessary on older GPUs and needs the dynamic loss scaler to prevent gradient underflow.

Limitations and common mistakes

DeepSpeed solves memory problems but introduces operational complexity that catches many teams. Most failures come from configuration mismatches, not from the algorithms themselves.

  • Offloading is slow: moving optimizer steps to the CPU can cut throughput severalfold, so it suits capacity-limited experiments rather than production pretraining.
  • Checkpoints are sharded: ZeRO stage 3 saves one file per rank, and a conversion script is needed to obtain a consolidated fp32 model for inference.
  • Batch size arithmetic must match: the global batch equals micro-batch size times accumulation steps times data-parallel ranks, and inconsistent values stop initialization.
  • Custom kernels need compatible builds: some fused operators compile at first use and fail when CUDA, compiler or PyTorch versions disagree.
  • Network bandwidth limits scaling: stage 3 on nodes linked by ordinary Ethernet can be slower than a smaller model on fewer GPUs.

A frequent misconception is that DeepSpeed makes training faster by default. Its primary gain is capacity; speed improves only when the larger effective batch or reduced memory pressure outweighs the added communication.

The idea in brief

Training a large AI model is like moving a grand piano: one person cannot carry it, but a team can if everyone takes a corner. DeepSpeed is the plan that tells each GPU which part of the load to carry, so no single card has to hold everything.

Normally every GPU keeps a full copy of the model and its bookkeeping, which wastes memory. DeepSpeed splits those copies into pieces, hands each GPU one piece and passes pieces around only when they are needed. The key takeaway is that DeepSpeed lets ordinary hardware train models that would otherwise not fit, at the cost of extra coordination between machines.

A practical scenario

A research team at a university wants to fine-tune a 13-billion-parameter language model on legal documents. Their cluster has two nodes with eight 40 GB GPUs each. With plain data parallelism, the roughly 208 GB of training state cannot fit on any single card, and the job fails with an out-of-memory error at the first optimizer step.

The engineers add a DeepSpeed configuration with ZeRO stage 3, bf16 precision and activation checkpointing, set a micro-batch of 2 with 8 accumulation steps, and launch the existing Hugging Face training script unchanged. Across 16 GPUs, each card now holds about 13 GB of sharded state, leaving room for activations at a 4,096-token context.

The fine-tuning finishes in about two days. Before deployment, the team converts the sharded checkpoint into a single fp32 file. The outcome is a working domain model trained on existing hardware, without purchasing larger accelerators.