FSDP, short for fully sharded data parallel, is a distributed training technique that splits a model’s parameters, gradients and optimizer states across all participating GPUs, so that each device stores only a shard of the full training state. In AI infrastructure, it allows teams to train models that are far too large to fit on a single accelerator while keeping the simple programming model of data parallelism.
FSDP is best known as the implementation in the PyTorch framework, and it applies the same idea as the ZeRO stage 3 optimization: instead of replicating the whole model on every GPU, the framework gathers parameters just in time for computation and frees them immediately afterward. The result is a large reduction in per-GPU memory at the cost of extra communication.
How FSDP shards and reassembles a model
FSDP works by dividing the model into units, usually transformer blocks, and storing each unit as a flat parameter tensor that is split evenly across the GPUs in the sharding group. Each GPU keeps only its own slice of parameters, the matching slice of gradients and the matching slice of optimizer state, such as the momentum and variance terms of the Adam optimizer.
During the forward pass, FSDP runs an all-gather for one unit to rebuild its full parameters, computes that layer, and then discards the gathered copy. The backward pass repeats the all-gather, computes gradients, and then performs a reduce-scatter, which sums gradients across GPUs and leaves each device with only the shard it owns. The optimizer step then updates each shard locally, without any further communication.
The table below summarizes the collective operations that FSDP issues in each training phase.
| Phase | Collective operation | Purpose |
|---|---|---|
| Forward pass | All-gather per unit | Rebuild full parameters for computation |
| Backward pass | All-gather per unit | Rebuild parameters to compute gradients |
| Gradient synchronization | Reduce-scatter per unit | Sum gradients and keep only the local shard |
| Optimizer step | None | Update the local parameter shard |
Because the gathered parameters live in memory only while one unit is being processed, peak memory depends on the size of the largest unit plus activations, not on the size of the whole model. FSDP hides much of the communication cost through prefetching, which starts gathering the next unit while the current one is still computing.
Sharding strategies and key settings
FSDP offers several sharding strategies that set how much state is split and therefore how memory is traded against network traffic. Choosing among them depends mainly on model size, the number of GPUs and the bandwidth between nodes.
| Strategy | What is sharded | Memory saving | Communication cost |
|---|---|---|---|
| Full shard | Parameters, gradients, optimizer states | Highest | Highest |
| Shard grad op | Gradients and optimizer states | Moderate | Lower than full shard |
| Hybrid shard | Full shard inside a node, replicate across nodes | High | Mostly fast intra-node traffic |
| No shard | Nothing | None | Same as classic data parallelism |
Beyond the strategy, several settings strongly affect throughput and stability. The wrapping policy decides which modules become sharding units; wrapping each transformer block is the usual choice. Mixed precision keeps a full-precision master copy of the weights while computing and communicating in bfloat16 or float16. CPU offloading moves parameter shards to host memory, which saves GPU memory but slows training considerably.
Newer versions of the PyTorch implementation, often called FSDP2, shard each parameter individually as a distributed tensor rather than flattening whole units. This design simplifies composition with tensor parallelism and makes checkpoints easier to save and reload with different GPU counts.
FSDP compared with other parallelism methods
FSDP is a memory-efficient form of data parallelism: every GPU still processes a different slice of the training batch, but no GPU holds the full model state permanently. This sets it apart from methods that split the computation itself across devices.
| Method | What is split | Main benefit | Main drawback |
|---|---|---|---|
| Data parallelism | Training batch only | Simple and fast for small models | Full model copy on every GPU |
| FSDP | Batch plus all training state | Fits very large models with little code change | Heavy all-gather traffic |
| Tensor parallelism | Individual matrix operations | Splits layers too large for one GPU | Needs very fast links such as NVLink |
| Pipeline parallelism | Groups of consecutive layers | Scales across many nodes | Pipeline bubbles and scheduling complexity |
In practice, the largest training runs combine methods into multi-dimensional parallelism: FSDP across data-parallel groups, tensor parallelism inside a node and sometimes pipeline or context parallelism across nodes. For models up to several tens of billions of parameters, FSDP alone is often sufficient and far easier to operate.
When to use FSDP and how to tune it
FSDP is the right choice when a model, together with its optimizer state, does not fit in the memory of one GPU but each layer still does. A rough rule is that mixed-precision training with Adam needs about 16 bytes per parameter for weights, gradients and optimizer state, before activations, so a 7-billion-parameter model already exceeds a single 80 GB accelerator.
| Situation | Recommended setting | Reason |
|---|---|---|
| Model fits on one GPU | Classic data parallelism | Avoids unnecessary communication |
| Model too large, single node | Full shard | Maximum memory saving over fast links |
| Many nodes, slower interconnect | Hybrid shard | Keeps most traffic inside each node |
| Activations dominate memory | Activation checkpointing | Recomputes activations instead of storing them |
| GPU memory still insufficient | CPU offloading | Last resort that trades speed for capacity |
Tuning usually starts with the wrapping granularity and the micro-batch size, then checks model FLOPs utilization to confirm that GPUs are computing rather than waiting for the network. Larger units reduce the number of collectives but raise peak memory, so the best balance is found empirically.
Limitations and common mistakes
FSDP is bandwidth-bound on slow networks, because every training step moves the full parameter set across the sharding group at least twice. Most problems in practice come from configuration rather than from the method itself.
- Wrapping the whole model as one unit gathers every parameter at once and removes the memory benefit.
- Saving full checkpoints on one rank can exhaust host memory; sharded or distributed checkpoints are safer.
- Mixing precision settings carelessly can cause loss spikes, so gradient reduction precision deserves attention.
- Ignoring interconnect limits across nodes leads to low utilization, where hybrid sharding often helps.
- Expecting FSDP to split a single oversized layer fails, because each unit must still fit when gathered.
Understanding it simply
Imagine a team of eight people assembling a huge jigsaw puzzle, but no single table is big enough to hold the whole picture. Each person keeps one eighth of the pieces in a drawer. When the team works on one area, everyone briefly lays out their pieces for that area, finishes it, and puts the pieces back.
FSDP does the same for AI models. Each graphics card stores only a slice of the model, and the slices are shared briefly whenever a part of the model is needed. The key takeaway is that FSDP lets ordinary GPU clusters train models that are too big for any single card, paid for with extra data traffic between the cards.
Applied in practice
A research team wants to fine-tune a 13-billion-parameter language model on legal documents using one server with eight GPUs of 80 GB each. With classic data parallelism, each GPU would need roughly 200 GB for weights, gradients and Adam states, so training fails with out-of-memory errors.
The team switches to FSDP with full sharding, wraps each transformer block as a unit, enables bfloat16 mixed precision and turns on activation checkpointing. Each GPU now holds about one eighth of the training state, roughly 26 GB, leaving room for activations at a useful batch size.
Training runs at a stable utilization of around 45 percent of peak FLOPs, and checkpoints are saved in sharded form so that later runs can resume on a different number of GPUs. The outcome is a full fine-tune on a single node, without rewriting the model code for tensor or pipeline parallelism.