FSDP

Also known as: Fully sharded data parallel

On this page 7

FSDP, short for fully sharded data parallel, is a distributed training technique that splits a model’s parameters, gradients and optimizer states across all participating GPUs, so that each device stores only a shard of the full training state. In AI infrastructure, it allows teams to train models that are far too large to fit on a single accelerator while keeping the simple programming model of data parallelism.

FSDP is best known as the implementation in the PyTorch framework, and it applies the same idea as the ZeRO stage 3 optimization: instead of replicating the whole model on every GPU, the framework gathers parameters just in time for computation and frees them immediately afterward. The result is a large reduction in per-GPU memory at the cost of extra communication.

How FSDP shards and reassembles a model

FSDP works by dividing the model into units, usually transformer blocks, and storing each unit as a flat parameter tensor that is split evenly across the GPUs in the sharding group. Each GPU keeps only its own slice of parameters, the matching slice of gradients and the matching slice of optimizer state, such as the momentum and variance terms of the Adam optimizer.

During the forward pass, FSDP runs an all-gather for one unit to rebuild its full parameters, computes that layer, and then discards the gathered copy. The backward pass repeats the all-gather, computes gradients, and then performs a reduce-scatter, which sums gradients across GPUs and leaves each device with only the shard it owns. The optimizer step then updates each shard locally, without any further communication.

The table below summarizes the collective operations that FSDP issues in each training phase.

PhaseCollective operationPurpose
Forward passAll-gather per unitRebuild full parameters for computation
Backward passAll-gather per unitRebuild parameters to compute gradients
Gradient synchronizationReduce-scatter per unitSum gradients and keep only the local shard
Optimizer stepNoneUpdate the local parameter shard
Communication in one FSDP training step

Because the gathered parameters live in memory only while one unit is being processed, peak memory depends on the size of the largest unit plus activations, not on the size of the whole model. FSDP hides much of the communication cost through prefetching, which starts gathering the next unit while the current one is still computing.

Sharding strategies and key settings

FSDP offers several sharding strategies that set how much state is split and therefore how memory is traded against network traffic. Choosing among them depends mainly on model size, the number of GPUs and the bandwidth between nodes.

StrategyWhat is shardedMemory savingCommunication cost
Full shardParameters, gradients, optimizer statesHighestHighest
Shard grad opGradients and optimizer statesModerateLower than full shard
Hybrid shardFull shard inside a node, replicate across nodesHighMostly fast intra-node traffic
No shardNothingNoneSame as classic data parallelism
FSDP sharding strategies

Beyond the strategy, several settings strongly affect throughput and stability. The wrapping policy decides which modules become sharding units; wrapping each transformer block is the usual choice. Mixed precision keeps a full-precision master copy of the weights while computing and communicating in bfloat16 or float16. CPU offloading moves parameter shards to host memory, which saves GPU memory but slows training considerably.

Newer versions of the PyTorch implementation, often called FSDP2, shard each parameter individually as a distributed tensor rather than flattening whole units. This design simplifies composition with tensor parallelism and makes checkpoints easier to save and reload with different GPU counts.

FSDP compared with other parallelism methods

FSDP is a memory-efficient form of data parallelism: every GPU still processes a different slice of the training batch, but no GPU holds the full model state permanently. This sets it apart from methods that split the computation itself across devices.

MethodWhat is splitMain benefitMain drawback
Data parallelismTraining batch onlySimple and fast for small modelsFull model copy on every GPU
FSDPBatch plus all training stateFits very large models with little code changeHeavy all-gather traffic
Tensor parallelismIndividual matrix operationsSplits layers too large for one GPUNeeds very fast links such as NVLink
Pipeline parallelismGroups of consecutive layersScales across many nodesPipeline bubbles and scheduling complexity
How FSDP relates to other distributed training methods

In practice, the largest training runs combine methods into multi-dimensional parallelism: FSDP across data-parallel groups, tensor parallelism inside a node and sometimes pipeline or context parallelism across nodes. For models up to several tens of billions of parameters, FSDP alone is often sufficient and far easier to operate.

When to use FSDP and how to tune it

FSDP is the right choice when a model, together with its optimizer state, does not fit in the memory of one GPU but each layer still does. A rough rule is that mixed-precision training with Adam needs about 16 bytes per parameter for weights, gradients and optimizer state, before activations, so a 7-billion-parameter model already exceeds a single 80 GB accelerator.

SituationRecommended settingReason
Model fits on one GPUClassic data parallelismAvoids unnecessary communication
Model too large, single nodeFull shardMaximum memory saving over fast links
Many nodes, slower interconnectHybrid shardKeeps most traffic inside each node
Activations dominate memoryActivation checkpointingRecomputes activations instead of storing them
GPU memory still insufficientCPU offloadingLast resort that trades speed for capacity
Choosing an FSDP configuration

Tuning usually starts with the wrapping granularity and the micro-batch size, then checks model FLOPs utilization to confirm that GPUs are computing rather than waiting for the network. Larger units reduce the number of collectives but raise peak memory, so the best balance is found empirically.

Limitations and common mistakes

FSDP is bandwidth-bound on slow networks, because every training step moves the full parameter set across the sharding group at least twice. Most problems in practice come from configuration rather than from the method itself.

  • Wrapping the whole model as one unit gathers every parameter at once and removes the memory benefit.
  • Saving full checkpoints on one rank can exhaust host memory; sharded or distributed checkpoints are safer.
  • Mixing precision settings carelessly can cause loss spikes, so gradient reduction precision deserves attention.
  • Ignoring interconnect limits across nodes leads to low utilization, where hybrid sharding often helps.
  • Expecting FSDP to split a single oversized layer fails, because each unit must still fit when gathered.

Understanding it simply

Imagine a team of eight people assembling a huge jigsaw puzzle, but no single table is big enough to hold the whole picture. Each person keeps one eighth of the pieces in a drawer. When the team works on one area, everyone briefly lays out their pieces for that area, finishes it, and puts the pieces back.

FSDP does the same for AI models. Each graphics card stores only a slice of the model, and the slices are shared briefly whenever a part of the model is needed. The key takeaway is that FSDP lets ordinary GPU clusters train models that are too big for any single card, paid for with extra data traffic between the cards.

Applied in practice

A research team wants to fine-tune a 13-billion-parameter language model on legal documents using one server with eight GPUs of 80 GB each. With classic data parallelism, each GPU would need roughly 200 GB for weights, gradients and Adam states, so training fails with out-of-memory errors.

The team switches to FSDP with full sharding, wraps each transformer block as a unit, enables bfloat16 mixed precision and turns on activation checkpointing. Each GPU now holds about one eighth of the training state, roughly 26 GB, leaving room for activations at a useful batch size.

Training runs at a stable utilization of around 45 percent of peak FLOPs, and checkpoints are saved in sharded form so that later runs can resume on a different number of GPUs. The outcome is a full fine-tune on a single node, without rewriting the model code for tensor or pipeline parallelism.