Model parallelism is a family of techniques that split a single neural network across several accelerators, so that each GPU or TPU holds and computes only part of the model. In AI infrastructure, it is the standard answer when a model’s parameters, activations and optimizer state exceed the memory of one device, which is the case for virtually every large language model trained or served today.
Model parallelism contrasts with data parallelism, where every device keeps a full copy of the model and processes a different slice of the data. The two are usually combined: model parallelism makes the model fit, and data parallelism adds throughput. The price is communication between devices, which turns interconnect bandwidth into a first-class design constraint.
How model parallelism divides a network
A network can be cut along two basic axes. Inter-layer partitioning assigns whole consecutive layers to different devices, which is the idea behind pipeline parallelism. Intra-layer partitioning splits the weight matrices inside each layer, so that several devices compute parts of the same matrix multiplication, which is the idea behind tensor parallelism.
Both approaches require devices to exchange intermediate results. In inter-layer splits, activations flow forward and gradients flow backward between neighboring stages. In intra-layer splits, devices must combine partial results with collective operations such as all-reduce or all-gather after nearly every layer, which makes this style far more sensitive to link speed.
Memory pressure is the main driver. A model with 70 billion parameters needs about 140 GB just for 16-bit weights, and training with the Adam optimizer in mixed precision typically requires roughly 16 bytes per parameter before activations are counted, which exceeds the memory of any single current accelerator.
Main forms of model parallelism
Several specialized forms have emerged, each splitting a different dimension of the computation. The following table summarizes what each one partitions and where its costs appear.
| Form | What is split | Communication pattern | Main cost |
|---|---|---|---|
| Tensor parallelism | Weight matrices within a layer | All-reduce per layer | Needs very fast links |
| Pipeline parallelism | Groups of consecutive layers | Point-to-point activations | Idle pipeline bubbles |
| Expert parallelism | Experts of a mixture-of-experts layer | All-to-all token routing | Load imbalance between experts |
| Sequence parallelism | Activations along the sequence | All-gather and reduce-scatter | Extra collective operations |
| Context parallelism | Long input contexts across devices | Ring exchange of keys and values | Latency on very long sequences |
Sharded data parallel methods such as ZeRO and FSDP are sometimes grouped with model parallelism because they also split parameters and optimizer state across devices. Strictly, they remain data parallel: each device still computes the full layer after gathering the parameters it needs, trading extra communication for memory savings.
Combining strategies in large training runs
Frontier-scale training uses three-dimensional or higher-dimensional parallelism, which nests several strategies so each one runs over the network layer that suits it. The guiding principle is to place the most communication-intensive split on the fastest links.
| Strategy | Usual scope | Reason for placement |
|---|---|---|
| Tensor parallelism | Within one server, often 8 GPUs | Uses NVLink-class bandwidth |
| Pipeline parallelism | Across servers in a rack or pod | Tolerates slower network links |
| Data parallelism | Across the whole cluster | Gradient sync can overlap compute |
| Expert parallelism | Within or across servers | Depends on routing traffic volume |
The degree of each dimension multiplies: for example, tensor degree 8, pipeline degree 4 and data degree 64 together occupy 2,048 accelerators. Choosing these degrees is an optimization problem balancing memory per device, communication volume and pipeline efficiency, and teams commonly measure the result with model FLOPs utilization.
Model parallelism in inference
Serving also needs model parallelism once weights and the KV cache no longer fit on one device. Inference servers such as vLLM shard large models with tensor parallelism inside a node to cut per-token latency, and add pipeline parallelism across nodes when a single node is still too small.
The trade-offs differ from training. Inference has no backward pass, so pipeline bubbles matter less with continuous batching, but every decoding step crosses device boundaries, so interconnect latency directly raises time per output token. Mixture-of-experts models add expert parallelism so that each device serves a subset of experts.
Choosing and tuning a parallel layout
No single layout is optimal; the right choice depends on model size, hardware topology and workload. The decision table below captures common starting points used by infrastructure teams.
| Situation | Recommended approach | Why |
|---|---|---|
| Model fits on one GPU | Data parallelism only | Avoids unnecessary communication |
| Fits in one server | Tensor parallelism within the node | Fast links keep overhead low |
| Exceeds one server | Add pipeline parallelism across nodes | Point-to-point traffic suits slower links |
| Mixture-of-experts model | Add expert parallelism | Spreads experts and their memory |
| Very long contexts | Add sequence or context parallelism | Splits activation memory by length |
Common mistakes include stretching tensor parallelism across nodes, which usually collapses throughput, using too few micro-batches so that pipeline bubbles dominate, and splitting layers unevenly so one stage becomes the bottleneck. Profiling a short run before committing to a full job is far cheaper than discovering the imbalance mid-training.
Limitations and engineering risks
- Communication overhead grows with the number of splits and can erase the benefit of extra devices.
- Fault tolerance weakens, because one failed device stalls the whole model replica.
- Implementation complexity is high; frameworks such as Megatron-LM and DeepSpeed hide much of it but constrain model code.
- Checkpoint resharding is required when the parallel layout changes between training and serving.
The short version
Model parallelism means sharing one giant AI model across many chips because no single chip has enough memory to hold it. Each chip keeps a piece of the model and does its part of the math.
Think of a large book translated by a team. Each translator handles certain chapters, or even certain paragraphs on every page, and they pass their work along. The team finishes a book no single person could manage, but they must coordinate constantly, and slow coordination wastes much of the gain.
Where it makes a difference
A research group wants to fine-tune a 70-billion-parameter language model on a cluster of servers, each with eight GPUs of 80 GB. A single GPU cannot even hold the 16-bit weights, so plain data parallelism is impossible.
The engineers apply tensor parallelism of degree 8 inside each server, where the GPUs share a high-bandwidth interconnect, and pipeline parallelism of degree 2 across pairs of servers. Data parallelism then replicates this 16-GPU unit across the remaining hardware. After tuning micro-batch counts to shrink pipeline bubbles, the job reaches stable utilization, and training that was previously impossible finishes in days.