Tensor parallelism is a model parallelism technique that splits individual weight matrices of a neural network layer across several accelerators, so each device computes only a slice of every matrix multiplication. In AI infrastructure, it is the standard way to run one layer on many GPUs at once when a model is too large or too slow for a single device.
Unlike data parallelism, which copies the whole model to every device, and pipeline parallelism, which assigns whole layers to different devices, tensor parallelism divides the work inside each layer. The price is frequent collective communication, which is why it is normally confined to GPUs joined by very fast interconnects within one server.
How tensor parallelism splits a layer
Tensor parallelism works by partitioning a weight matrix along its columns or rows and giving each GPU one shard. Each device multiplies its local activations by its shard, and a collective operation such as all-reduce or all-gather combines the partial results into the full output.
The widely used scheme popularized by Megatron-LM pairs the two partitioning styles inside a transformer block. In the multilayer perceptron, the first matrix is split by columns and the second by rows, so the intermediate activation never needs to be gathered and only one all-reduce per block is required in the forward pass. In self-attention, the attention heads are distributed across devices, and the output projection is split by rows.
The table below compares the two basic ways to split a linear layer.
| Split type | What each GPU holds | Communication needed | Typical position |
|---|---|---|---|
| Column-parallel | A slice of output features | All-gather if full output is needed | First MLP matrix, QKV projection |
| Row-parallel | A slice of input features | All-reduce to sum partial outputs | Second MLP matrix, attention output |
| Vocabulary-parallel | A slice of the embedding table | All-reduce or fused cross-entropy | Input embedding, output logits |
Because every GPU holds only a fraction of the weights, memory per device falls roughly in proportion to the tensor parallel degree, and the matching shard of the KV cache shrinks as well during inference.
Communication cost and hardware requirements
The main limitation of tensor parallelism is that it communicates in every layer, both in the forward pass and in the backward pass during training. Its scaling efficiency therefore depends directly on interconnect bandwidth and latency between the participating GPUs.
Inside a server, links such as NVLink and NVSwitch provide hundreds of gigabytes per second per GPU, which keeps the overhead manageable. Across servers, even fast InfiniBand or Ethernet fabrics are usually an order of magnitude slower, so tensor parallel groups rarely span nodes. The common practical limit is a degree of 2, 4 or 8, matching the GPUs in one server.
| Degree | Memory per GPU | Communication share | Typical use |
|---|---|---|---|
| 1 | Full layer weights | None | Models that fit on one GPU |
| 2 | About half | Low | Mid-size models, latency gains |
| 4 | About a quarter | Moderate | Large models on one server |
| 8 | About an eighth | Significant | Very large models on NVLink servers |
| 16 or more | Smaller still | Often dominant | Rare; needs multi-node NVLink domains |
Engineers reduce the cost by overlapping communication with computation, fusing collectives into kernels and adding sequence parallelism, which splits normalization and dropout along the sequence dimension to avoid redundant activation memory.
How tensor parallelism compares with other strategies
Tensor parallelism is one dimension of a multidimensional parallel strategy, and large training runs combine it with others. Each approach divides a different quantity, so each has a distinct communication pattern and memory profile.
| Strategy | What is divided | Main communication | Key drawback |
|---|---|---|---|
| Tensor parallelism | Matrices inside each layer | All-reduce every layer | Needs very fast links |
| Pipeline parallelism | Groups of consecutive layers | Point-to-point activations | Pipeline bubbles idle GPUs |
| Data parallelism | Training batch | Gradient all-reduce per step | Full model copy per GPU |
| FSDP and ZeRO | Parameters, gradients, optimizer state | All-gather and reduce-scatter | Extra traffic per layer |
| Expert parallelism | Mixture-of-experts experts | All-to-all token routing | Load imbalance across experts |
| Context parallelism | Long input sequences | Ring exchange of keys and values | Complex attention kernels |
A typical arrangement, often called 3D parallelism, uses tensor parallelism inside each server, pipeline parallelism across a few servers and data parallelism across the remaining replicas. This layout places the most communication-heavy dimension on the fastest links.
Tensor parallelism in inference serving
For inference, tensor parallelism is the usual way to lower per-token latency for models that exceed one GPU. Because every GPU works on every token, time per output token falls as the degree rises, while pipeline parallelism mainly raises throughput and leaves single-request latency largely unchanged.
Serving engines such as vLLM, TensorRT-LLM and SGLang expose a tensor parallel size setting that must divide the number of attention heads evenly. The decision table below shows typical choices.
| Situation | Recommended layout | Reason |
|---|---|---|
| Model fits on one GPU | Tensor parallel size 1, more replicas | Avoids communication overhead |
| Model fits on one server | Tensor parallelism across its GPUs | Lowest latency over fast links |
| Model exceeds one server | Tensor within node, pipeline across nodes | Keeps all-reduce off slow links |
| Throughput-focused batch jobs | Smaller degree, many replicas | Higher total tokens per second |
Limitations and common mistakes
Tensor parallelism delivers diminishing returns once communication time approaches computation time, and several configuration errors are common in practice.
- Spanning slow interconnects: placing one tensor parallel group across servers often makes the model slower than a smaller degree would.
- Choosing an indivisible degree: the number of attention heads and hidden dimensions must divide evenly by the degree.
- Ignoring small batch effects: at small batch sizes each shard multiplication becomes too small to use the GPU efficiently.
- Over-sharding small models: a model that fits on one GPU usually gains more from extra replicas than from splitting.
- Assuming identical numerics: changing the degree alters reduction order, so outputs can differ slightly in floating point.
The decisive rule is to use the smallest degree that fits the model and meets the latency target, and to keep each tensor parallel group within one high-bandwidth domain.
The idea in brief
Tensor parallelism means several graphics processors share the work of every single step in a large AI model. Instead of one chip multiplying a giant table of numbers, the table is cut into strips and each chip handles one strip at the same time.
Imagine a team of accountants splitting a huge spreadsheet by columns, then adding up their subtotals together. The work finishes sooner, but they must talk after every page. The main lesson is that this only pays off when the helpers sit very close together and can exchange results almost instantly.
A practical scenario
An engineering team needs to serve a 70-billion-parameter language model for an internal assistant. In 16-bit precision the weights alone need about 140 GB, which does not fit on a single 80 GB GPU, so the team deploys the model on one server with eight NVLink-connected GPUs and a tensor parallel size of 8.
Each GPU now holds roughly 18 GB of weights, leaving ample memory for the KV cache and continuous batching. Tests show that a tensor parallel size of 4 on two separate replicas gives higher total throughput, while size 8 gives the lowest time per output token.
Because the assistant is interactive, the team keeps the eight-way layout for latency and adds a second server only when demand grows, rather than stretching one tensor parallel group across both servers.