Model parallelism

On this page 8

Model parallelism is a family of techniques that split a single neural network across several accelerators, so that each GPU or TPU holds and computes only part of the model. In AI infrastructure, it is the standard answer when a model’s parameters, activations and optimizer state exceed the memory of one device, which is the case for virtually every large language model trained or served today.

Model parallelism contrasts with data parallelism, where every device keeps a full copy of the model and processes a different slice of the data. The two are usually combined: model parallelism makes the model fit, and data parallelism adds throughput. The price is communication between devices, which turns interconnect bandwidth into a first-class design constraint.

How model parallelism divides a network

A network can be cut along two basic axes. Inter-layer partitioning assigns whole consecutive layers to different devices, which is the idea behind pipeline parallelism. Intra-layer partitioning splits the weight matrices inside each layer, so that several devices compute parts of the same matrix multiplication, which is the idea behind tensor parallelism.

Both approaches require devices to exchange intermediate results. In inter-layer splits, activations flow forward and gradients flow backward between neighboring stages. In intra-layer splits, devices must combine partial results with collective operations such as all-reduce or all-gather after nearly every layer, which makes this style far more sensitive to link speed.

Memory pressure is the main driver. A model with 70 billion parameters needs about 140 GB just for 16-bit weights, and training with the Adam optimizer in mixed precision typically requires roughly 16 bytes per parameter before activations are counted, which exceeds the memory of any single current accelerator.

Main forms of model parallelism

Several specialized forms have emerged, each splitting a different dimension of the computation. The following table summarizes what each one partitions and where its costs appear.

FormWhat is splitCommunication patternMain cost
Tensor parallelismWeight matrices within a layerAll-reduce per layerNeeds very fast links
Pipeline parallelismGroups of consecutive layersPoint-to-point activationsIdle pipeline bubbles
Expert parallelismExperts of a mixture-of-experts layerAll-to-all token routingLoad imbalance between experts
Sequence parallelismActivations along the sequenceAll-gather and reduce-scatterExtra collective operations
Context parallelismLong input contexts across devicesRing exchange of keys and valuesLatency on very long sequences
Forms of model parallelism compared

Sharded data parallel methods such as ZeRO and FSDP are sometimes grouped with model parallelism because they also split parameters and optimizer state across devices. Strictly, they remain data parallel: each device still computes the full layer after gathering the parameters it needs, trading extra communication for memory savings.

Combining strategies in large training runs

Frontier-scale training uses three-dimensional or higher-dimensional parallelism, which nests several strategies so each one runs over the network layer that suits it. The guiding principle is to place the most communication-intensive split on the fastest links.

StrategyUsual scopeReason for placement
Tensor parallelismWithin one server, often 8 GPUsUses NVLink-class bandwidth
Pipeline parallelismAcross servers in a rack or podTolerates slower network links
Data parallelismAcross the whole clusterGradient sync can overlap compute
Expert parallelismWithin or across serversDepends on routing traffic volume
Typical placement of parallelism in a cluster

The degree of each dimension multiplies: for example, tensor degree 8, pipeline degree 4 and data degree 64 together occupy 2,048 accelerators. Choosing these degrees is an optimization problem balancing memory per device, communication volume and pipeline efficiency, and teams commonly measure the result with model FLOPs utilization.

Model parallelism in inference

Serving also needs model parallelism once weights and the KV cache no longer fit on one device. Inference servers such as vLLM shard large models with tensor parallelism inside a node to cut per-token latency, and add pipeline parallelism across nodes when a single node is still too small.

The trade-offs differ from training. Inference has no backward pass, so pipeline bubbles matter less with continuous batching, but every decoding step crosses device boundaries, so interconnect latency directly raises time per output token. Mixture-of-experts models add expert parallelism so that each device serves a subset of experts.

Choosing and tuning a parallel layout

No single layout is optimal; the right choice depends on model size, hardware topology and workload. The decision table below captures common starting points used by infrastructure teams.

SituationRecommended approachWhy
Model fits on one GPUData parallelism onlyAvoids unnecessary communication
Fits in one serverTensor parallelism within the nodeFast links keep overhead low
Exceeds one serverAdd pipeline parallelism across nodesPoint-to-point traffic suits slower links
Mixture-of-experts modelAdd expert parallelismSpreads experts and their memory
Very long contextsAdd sequence or context parallelismSplits activation memory by length
Decision guide for model parallelism

Common mistakes include stretching tensor parallelism across nodes, which usually collapses throughput, using too few micro-batches so that pipeline bubbles dominate, and splitting layers unevenly so one stage becomes the bottleneck. Profiling a short run before committing to a full job is far cheaper than discovering the imbalance mid-training.

Limitations and engineering risks

  • Communication overhead grows with the number of splits and can erase the benefit of extra devices.
  • Fault tolerance weakens, because one failed device stalls the whole model replica.
  • Implementation complexity is high; frameworks such as Megatron-LM and DeepSpeed hide much of it but constrain model code.
  • Checkpoint resharding is required when the parallel layout changes between training and serving.

The short version

Model parallelism means sharing one giant AI model across many chips because no single chip has enough memory to hold it. Each chip keeps a piece of the model and does its part of the math.

Think of a large book translated by a team. Each translator handles certain chapters, or even certain paragraphs on every page, and they pass their work along. The team finishes a book no single person could manage, but they must coordinate constantly, and slow coordination wastes much of the gain.

Where it makes a difference

A research group wants to fine-tune a 70-billion-parameter language model on a cluster of servers, each with eight GPUs of 80 GB. A single GPU cannot even hold the 16-bit weights, so plain data parallelism is impossible.

The engineers apply tensor parallelism of degree 8 inside each server, where the GPUs share a high-bandwidth interconnect, and pipeline parallelism of degree 2 across pairs of servers. Data parallelism then replicates this 16-GPU unit across the remaining hardware. After tuning micro-batch counts to shrink pipeline bubbles, the job reaches stable utilization, and training that was previously impossible finishes in days.