Tensor parallelism

On this page 7

Tensor parallelism is a model parallelism technique that splits individual weight matrices of a neural network layer across several accelerators, so each device computes only a slice of every matrix multiplication. In AI infrastructure, it is the standard way to run one layer on many GPUs at once when a model is too large or too slow for a single device.

Unlike data parallelism, which copies the whole model to every device, and pipeline parallelism, which assigns whole layers to different devices, tensor parallelism divides the work inside each layer. The price is frequent collective communication, which is why it is normally confined to GPUs joined by very fast interconnects within one server.

How tensor parallelism splits a layer

Tensor parallelism works by partitioning a weight matrix along its columns or rows and giving each GPU one shard. Each device multiplies its local activations by its shard, and a collective operation such as all-reduce or all-gather combines the partial results into the full output.

The widely used scheme popularized by Megatron-LM pairs the two partitioning styles inside a transformer block. In the multilayer perceptron, the first matrix is split by columns and the second by rows, so the intermediate activation never needs to be gathered and only one all-reduce per block is required in the forward pass. In self-attention, the attention heads are distributed across devices, and the output projection is split by rows.

The table below compares the two basic ways to split a linear layer.

Split typeWhat each GPU holdsCommunication neededTypical position
Column-parallelA slice of output featuresAll-gather if full output is neededFirst MLP matrix, QKV projection
Row-parallelA slice of input featuresAll-reduce to sum partial outputsSecond MLP matrix, attention output
Vocabulary-parallelA slice of the embedding tableAll-reduce or fused cross-entropyInput embedding, output logits
Column and row partitioning of a linear layer

Because every GPU holds only a fraction of the weights, memory per device falls roughly in proportion to the tensor parallel degree, and the matching shard of the KV cache shrinks as well during inference.

Communication cost and hardware requirements

The main limitation of tensor parallelism is that it communicates in every layer, both in the forward pass and in the backward pass during training. Its scaling efficiency therefore depends directly on interconnect bandwidth and latency between the participating GPUs.

Inside a server, links such as NVLink and NVSwitch provide hundreds of gigabytes per second per GPU, which keeps the overhead manageable. Across servers, even fast InfiniBand or Ethernet fabrics are usually an order of magnitude slower, so tensor parallel groups rarely span nodes. The common practical limit is a degree of 2, 4 or 8, matching the GPUs in one server.

DegreeMemory per GPUCommunication shareTypical use
1Full layer weightsNoneModels that fit on one GPU
2About halfLowMid-size models, latency gains
4About a quarterModerateLarge models on one server
8About an eighthSignificantVery large models on NVLink servers
16 or moreSmaller stillOften dominantRare; needs multi-node NVLink domains
Effect of the tensor parallel degree

Engineers reduce the cost by overlapping communication with computation, fusing collectives into kernels and adding sequence parallelism, which splits normalization and dropout along the sequence dimension to avoid redundant activation memory.

How tensor parallelism compares with other strategies

Tensor parallelism is one dimension of a multidimensional parallel strategy, and large training runs combine it with others. Each approach divides a different quantity, so each has a distinct communication pattern and memory profile.

StrategyWhat is dividedMain communicationKey drawback
Tensor parallelismMatrices inside each layerAll-reduce every layerNeeds very fast links
Pipeline parallelismGroups of consecutive layersPoint-to-point activationsPipeline bubbles idle GPUs
Data parallelismTraining batchGradient all-reduce per stepFull model copy per GPU
FSDP and ZeROParameters, gradients, optimizer stateAll-gather and reduce-scatterExtra traffic per layer
Expert parallelismMixture-of-experts expertsAll-to-all token routingLoad imbalance across experts
Context parallelismLong input sequencesRing exchange of keys and valuesComplex attention kernels
Parallelism strategies compared

A typical arrangement, often called 3D parallelism, uses tensor parallelism inside each server, pipeline parallelism across a few servers and data parallelism across the remaining replicas. This layout places the most communication-heavy dimension on the fastest links.

Tensor parallelism in inference serving

For inference, tensor parallelism is the usual way to lower per-token latency for models that exceed one GPU. Because every GPU works on every token, time per output token falls as the degree rises, while pipeline parallelism mainly raises throughput and leaves single-request latency largely unchanged.

Serving engines such as vLLM, TensorRT-LLM and SGLang expose a tensor parallel size setting that must divide the number of attention heads evenly. The decision table below shows typical choices.

SituationRecommended layoutReason
Model fits on one GPUTensor parallel size 1, more replicasAvoids communication overhead
Model fits on one serverTensor parallelism across its GPUsLowest latency over fast links
Model exceeds one serverTensor within node, pipeline across nodesKeeps all-reduce off slow links
Throughput-focused batch jobsSmaller degree, many replicasHigher total tokens per second
Choosing a serving layout

Limitations and common mistakes

Tensor parallelism delivers diminishing returns once communication time approaches computation time, and several configuration errors are common in practice.

  • Spanning slow interconnects: placing one tensor parallel group across servers often makes the model slower than a smaller degree would.
  • Choosing an indivisible degree: the number of attention heads and hidden dimensions must divide evenly by the degree.
  • Ignoring small batch effects: at small batch sizes each shard multiplication becomes too small to use the GPU efficiently.
  • Over-sharding small models: a model that fits on one GPU usually gains more from extra replicas than from splitting.
  • Assuming identical numerics: changing the degree alters reduction order, so outputs can differ slightly in floating point.

The decisive rule is to use the smallest degree that fits the model and meets the latency target, and to keep each tensor parallel group within one high-bandwidth domain.

The idea in brief

Tensor parallelism means several graphics processors share the work of every single step in a large AI model. Instead of one chip multiplying a giant table of numbers, the table is cut into strips and each chip handles one strip at the same time.

Imagine a team of accountants splitting a huge spreadsheet by columns, then adding up their subtotals together. The work finishes sooner, but they must talk after every page. The main lesson is that this only pays off when the helpers sit very close together and can exchange results almost instantly.

A practical scenario

An engineering team needs to serve a 70-billion-parameter language model for an internal assistant. In 16-bit precision the weights alone need about 140 GB, which does not fit on a single 80 GB GPU, so the team deploys the model on one server with eight NVLink-connected GPUs and a tensor parallel size of 8.

Each GPU now holds roughly 18 GB of weights, leaving ample memory for the KV cache and continuous batching. Tests show that a tensor parallel size of 4 on two separate replicas gives higher total throughput, while size 8 gives the lowest time per output token.

Because the assistant is interactive, the team keeps the eight-way layout for latency and adds a second server only when demand grows, rather than stretching one tensor parallel group across both servers.