Pipeline parallelism is a form of model parallelism in which a neural network is partitioned by layers into consecutive stages, each placed on a different accelerator, and input data is split into micro-batches that flow through the stages like items on an assembly line. In AI infrastructure, it is one of the core techniques for training and serving models whose weights and activations do not fit in the memory of a single GPU.
It is usually combined with tensor parallelism, which splits individual layers, and data parallelism, which replicates the whole model. The combination, often called 3D parallelism, is how frontier-scale language models are trained across thousands of GPUs.
How pipeline parallelism splits a model into stages
Pipeline parallelism assigns contiguous blocks of layers to separate devices. A 96-layer Transformer split across 8 GPUs might place 12 layers on each device. During the forward pass, each stage computes its layers and sends the resulting activations to the next stage; during the backward pass, gradients travel in the opposite direction.
Without further measures, only one stage works at a time while the others wait. The key idea is therefore to divide each mini-batch into many smaller micro-batches, so that stage 1 can process micro-batch 2 while stage 2 processes micro-batch 1. The more micro-batches in flight, the more stages stay busy simultaneously.
The table below lists the parameters that most strongly determine how a pipeline behaves.
| Parameter | Typical range | What it affects |
|---|---|---|
| Pipeline depth (stages) | 2 to 64 | Memory per device and bubble size |
| Micro-batches per step | 4 to several hundred | Utilization and activation memory |
| Micro-batch size | 1 to 16 sequences | Kernel efficiency per stage |
| Layers per stage | Balanced by compute cost | Load balance across devices |
| Interconnect bandwidth | Tens to hundreds of GB/s | Activation transfer latency |
The pipeline bubble and scheduling strategies
The pipeline bubble is the idle time at the start and end of each training step, when the pipeline is filling or draining. For a simple schedule with p stages and m micro-batches, the idle fraction is approximately (p − 1) / (m + p − 1). With 8 stages and 32 micro-batches, roughly 18 percent of device time is wasted, which is why practitioners keep m much larger than p.
Several schedules reduce the bubble or its memory cost. GPipe runs all forward passes, then all backward passes, which is simple but stores activations for every micro-batch. 1F1B (one forward, one backward), introduced with PipeDream, interleaves the two after a warm-up phase and caps activation memory at roughly p micro-batches. Interleaved 1F1B assigns several non-adjacent chunks of layers to each device, shrinking the bubble at the cost of more communication. Zero-bubble schedules split the backward pass into input-gradient and weight-gradient parts to fill idle slots almost completely.
| Schedule | Bubble size | Activation memory | Main trade-off |
|---|---|---|---|
| GPipe | Moderate | High, all micro-batches | Simple but memory-hungry |
| 1F1B | Moderate | Bounded by stage count | Standard default in training frameworks |
| Interleaved 1F1B | Smaller | Bounded, slightly higher | More point-to-point communication |
| Zero-bubble variants | Near zero | Similar to 1F1B | Complex implementation and tuning |
Pipeline parallelism compared with other parallelism strategies
Pipeline parallelism differs from other strategies in what it splits and what it communicates. Data parallelism splits the batch and synchronizes gradients with all-reduce; tensor parallelism splits matrix operations inside each layer and needs very fast links; pipeline parallelism splits the layer sequence and sends only activations between neighboring stages.
Because its communication is point-to-point and relatively light, pipeline parallelism works well across nodes connected by slower networks, while tensor parallelism is usually kept within a single server where NVLink or a similar fabric is available. Sharded approaches such as FSDP and ZeRO reduce memory by partitioning weights and optimizer state but still compute each layer on every device.
| Strategy | What is split | Communication pattern | Best placement |
|---|---|---|---|
| Data parallelism | Training batch | All-reduce of gradients | Any scale, any network |
| Tensor parallelism | Weights inside a layer | Frequent collectives per layer | Within one server |
| Pipeline parallelism | Sequence of layers | Point-to-point activations | Across servers |
| Expert parallelism | Mixture-of-experts experts | All-to-all token routing | Across servers with fast fabric |
| FSDP and ZeRO | Parameters and optimizer state | Gather and reduce-scatter | Data-parallel groups |
Configuring pipelines for training and inference
Good pipeline configuration starts with balancing compute across stages, because the slowest stage sets the pace for all others. Embedding and output layers, which can be large in language models, often need special placement or a smaller share of Transformer blocks on the same stage. Teams usually choose the smallest pipeline depth that fits memory, then raise micro-batch count to hide the bubble.
In inference, pipeline parallelism lets a model that exceeds one GPU’s memory be served across several devices or nodes. It adds per-token latency because each request crosses every stage, so serving systems prefer tensor parallelism inside a node and use pipeline stages mainly when a model spans nodes. Continuous batching keeps stages busy by feeding new requests into the pipeline as others finish.
| Situation | Recommended approach | Reason |
|---|---|---|
| Model fits on one GPU | Avoid pipeline parallelism | Data parallelism is simpler and faster |
| Model fits in one server | Tensor parallelism first | High-bandwidth links absorb collectives |
| Model spans many servers | Add pipeline stages across nodes | Light inter-node communication |
| Small global batch | Use interleaved or zero-bubble schedules | Few micro-batches enlarge the bubble |
| Latency-critical serving | Minimize pipeline depth | Each stage adds a network hop |
Limitations and common mistakes
Pipeline parallelism brings engineering complexity and efficiency losses that must be weighed against its memory benefit. The bubble never disappears entirely in simple schedules, uneven stages waste capacity, and debugging a failure that occurs only on one stage is considerably harder than in a single-device setup.
- Too few micro-batches: the bubble dominates and GPU utilization drops sharply.
- Unbalanced stages: one heavy stage throttles the whole pipeline.
- Ignoring activation memory: GPipe-style schedules can exhaust memory despite splitting weights.
- Excessive depth: each added stage increases latency and fill-and-drain overhead.
- Weight staleness: asynchronous variants such as early PipeDream can update weights inconsistently and need careful handling.
Explained without jargon
Pipeline parallelism is like a car assembly line for a giant AI model. The model is too big for one computer chip, so it is cut into sections, and each chip handles one section. Data passes from chip to chip, just as a car moves from station to station.
To keep every station busy, the work is split into many small batches. While the last station finishes one batch, the first is already starting the next. The more small batches flow through, the less time any chip sits idle.
A typical use case
A research team trains a 70-billion-parameter language model on a cluster of 64 GPUs spread across 8 servers, each with 8 GPUs and fast internal links. The model weights, gradients and optimizer state cannot fit inside a single server, so the team needs a combined approach.
They use tensor parallelism of 8 within each server and pipeline parallelism of 4 across servers, leaving data parallelism of 2 for the remaining dimension. Each training step processes 64 micro-batches with an interleaved 1F1B schedule, which keeps the estimated bubble below 5 percent.
After rebalancing the first and last stages, which carry the embedding and output layers, measured throughput rises by about 12 percent. The team reaches a model FLOPs utilization near 45 percent, a solid result for this scale, and completes training within the planned compute budget.