Pipeline parallelism

On this page 7

Pipeline parallelism is a form of model parallelism in which a neural network is partitioned by layers into consecutive stages, each placed on a different accelerator, and input data is split into micro-batches that flow through the stages like items on an assembly line. In AI infrastructure, it is one of the core techniques for training and serving models whose weights and activations do not fit in the memory of a single GPU.

It is usually combined with tensor parallelism, which splits individual layers, and data parallelism, which replicates the whole model. The combination, often called 3D parallelism, is how frontier-scale language models are trained across thousands of GPUs.

How pipeline parallelism splits a model into stages

Pipeline parallelism assigns contiguous blocks of layers to separate devices. A 96-layer Transformer split across 8 GPUs might place 12 layers on each device. During the forward pass, each stage computes its layers and sends the resulting activations to the next stage; during the backward pass, gradients travel in the opposite direction.

Without further measures, only one stage works at a time while the others wait. The key idea is therefore to divide each mini-batch into many smaller micro-batches, so that stage 1 can process micro-batch 2 while stage 2 processes micro-batch 1. The more micro-batches in flight, the more stages stay busy simultaneously.

The table below lists the parameters that most strongly determine how a pipeline behaves.

ParameterTypical rangeWhat it affects
Pipeline depth (stages)2 to 64Memory per device and bubble size
Micro-batches per step4 to several hundredUtilization and activation memory
Micro-batch size1 to 16 sequencesKernel efficiency per stage
Layers per stageBalanced by compute costLoad balance across devices
Interconnect bandwidthTens to hundreds of GB/sActivation transfer latency
Key parameters of a training pipeline

The pipeline bubble and scheduling strategies

The pipeline bubble is the idle time at the start and end of each training step, when the pipeline is filling or draining. For a simple schedule with p stages and m micro-batches, the idle fraction is approximately (p − 1) / (m + p − 1). With 8 stages and 32 micro-batches, roughly 18 percent of device time is wasted, which is why practitioners keep m much larger than p.

Several schedules reduce the bubble or its memory cost. GPipe runs all forward passes, then all backward passes, which is simple but stores activations for every micro-batch. 1F1B (one forward, one backward), introduced with PipeDream, interleaves the two after a warm-up phase and caps activation memory at roughly p micro-batches. Interleaved 1F1B assigns several non-adjacent chunks of layers to each device, shrinking the bubble at the cost of more communication. Zero-bubble schedules split the backward pass into input-gradient and weight-gradient parts to fill idle slots almost completely.

ScheduleBubble sizeActivation memoryMain trade-off
GPipeModerateHigh, all micro-batchesSimple but memory-hungry
1F1BModerateBounded by stage countStandard default in training frameworks
Interleaved 1F1BSmallerBounded, slightly higherMore point-to-point communication
Zero-bubble variantsNear zeroSimilar to 1F1BComplex implementation and tuning
Common pipeline schedules

Pipeline parallelism compared with other parallelism strategies

Pipeline parallelism differs from other strategies in what it splits and what it communicates. Data parallelism splits the batch and synchronizes gradients with all-reduce; tensor parallelism splits matrix operations inside each layer and needs very fast links; pipeline parallelism splits the layer sequence and sends only activations between neighboring stages.

Because its communication is point-to-point and relatively light, pipeline parallelism works well across nodes connected by slower networks, while tensor parallelism is usually kept within a single server where NVLink or a similar fabric is available. Sharded approaches such as FSDP and ZeRO reduce memory by partitioning weights and optimizer state but still compute each layer on every device.

StrategyWhat is splitCommunication patternBest placement
Data parallelismTraining batchAll-reduce of gradientsAny scale, any network
Tensor parallelismWeights inside a layerFrequent collectives per layerWithin one server
Pipeline parallelismSequence of layersPoint-to-point activationsAcross servers
Expert parallelismMixture-of-experts expertsAll-to-all token routingAcross servers with fast fabric
FSDP and ZeROParameters and optimizer stateGather and reduce-scatterData-parallel groups
How parallelism strategies divide the work

Configuring pipelines for training and inference

Good pipeline configuration starts with balancing compute across stages, because the slowest stage sets the pace for all others. Embedding and output layers, which can be large in language models, often need special placement or a smaller share of Transformer blocks on the same stage. Teams usually choose the smallest pipeline depth that fits memory, then raise micro-batch count to hide the bubble.

In inference, pipeline parallelism lets a model that exceeds one GPU’s memory be served across several devices or nodes. It adds per-token latency because each request crosses every stage, so serving systems prefer tensor parallelism inside a node and use pipeline stages mainly when a model spans nodes. Continuous batching keeps stages busy by feeding new requests into the pipeline as others finish.

SituationRecommended approachReason
Model fits on one GPUAvoid pipeline parallelismData parallelism is simpler and faster
Model fits in one serverTensor parallelism firstHigh-bandwidth links absorb collectives
Model spans many serversAdd pipeline stages across nodesLight inter-node communication
Small global batchUse interleaved or zero-bubble schedulesFew micro-batches enlarge the bubble
Latency-critical servingMinimize pipeline depthEach stage adds a network hop
Choosing a pipeline setup

Limitations and common mistakes

Pipeline parallelism brings engineering complexity and efficiency losses that must be weighed against its memory benefit. The bubble never disappears entirely in simple schedules, uneven stages waste capacity, and debugging a failure that occurs only on one stage is considerably harder than in a single-device setup.

  • Too few micro-batches: the bubble dominates and GPU utilization drops sharply.
  • Unbalanced stages: one heavy stage throttles the whole pipeline.
  • Ignoring activation memory: GPipe-style schedules can exhaust memory despite splitting weights.
  • Excessive depth: each added stage increases latency and fill-and-drain overhead.
  • Weight staleness: asynchronous variants such as early PipeDream can update weights inconsistently and need careful handling.

Explained without jargon

Pipeline parallelism is like a car assembly line for a giant AI model. The model is too big for one computer chip, so it is cut into sections, and each chip handles one section. Data passes from chip to chip, just as a car moves from station to station.

To keep every station busy, the work is split into many small batches. While the last station finishes one batch, the first is already starting the next. The more small batches flow through, the less time any chip sits idle.

A typical use case

A research team trains a 70-billion-parameter language model on a cluster of 64 GPUs spread across 8 servers, each with 8 GPUs and fast internal links. The model weights, gradients and optimizer state cannot fit inside a single server, so the team needs a combined approach.

They use tensor parallelism of 8 within each server and pipeline parallelism of 4 across servers, leaving data parallelism of 2 for the remaining dimension. Each training step processes 64 micro-batches with an interleaved 1F1B schedule, which keeps the estimated bubble below 5 percent.

After rebalancing the first and last stages, which carry the embedding and output layers, measured throughput rises by about 12 percent. The team reaches a model FLOPs utilization near 45 percent, a solid result for this scale, and completes training within the planned compute budget.