Sequence parallelism

On this page 7

Sequence parallelism is a distributed training technique that partitions activations along the sequence (token) dimension across several accelerators, so that each device stores and processes only a slice of every input sequence. In AI infrastructure it is used mainly to reduce activation memory in large transformer models, which makes longer context windows and larger micro-batches possible on the same hardware.

The term has two closely related uses. In the narrow Megatron-LM sense, it complements tensor parallelism by splitting the layers that tensor parallelism leaves replicated. In a broader sense, it covers any scheme that shards tokens, including attention-level approaches that are now often called context parallelism.

How sequence parallelism splits transformer activations

Sequence parallelism works by giving each device a contiguous chunk of tokens for the operations that act on tokens independently. Layer normalization, dropout and residual additions treat every position separately, so they can run on a shard of the sequence without any communication.

In the Megatron-LM design, tensor parallelism already splits the attention and MLP weight matrices across a group of GPUs, but the activations between those blocks remain fully replicated on every device. Sequence parallelism shards exactly those replicated regions. The usual all-reduce after a tensor-parallel block is replaced by a reduce-scatter, which leaves each device with its own token slice, and an all-gather restores the full sequence before the next tensor-parallel block.

Because one all-reduce is mathematically equivalent to a reduce-scatter followed by an all-gather, this form of sequence parallelism adds almost no extra communication volume. Its benefit is that activation memory for the non-tensor-parallel regions drops by a factor equal to the tensor-parallel group size.

The table below summarizes what happens to each part of a transformer layer.

Layer componentWithout sequence parallelismWith sequence parallelism
Layer normalizationFull sequence replicated on each GPUEach GPU normalizes its own token slice
Dropout and residualsActivations replicatedActivations sharded by tokens
Attention and MLP weightsSplit by tensor parallelismUnchanged, still split by tensor parallelism
Block boundary communicationAll-reduceReduce-scatter plus all-gather
Treatment of transformer layer components

Variants of sequence parallelism

Several variants of sequence parallelism exist, and they differ mainly in how attention is handled, since attention is the one operation where every token must see other tokens. The choice determines how far the technique scales with context length.

ApproachAttention handlingMain communicationTypical use
Megatron-styleAttention stays under tensor parallelismReduce-scatter and all-gatherPaired with tensor parallelism inside a node
Ring attentionKey and value blocks rotate around a ring of devicesPoint-to-point transfers overlapped with computeVery long contexts across many devices
All-to-all head splitTokens regrouped so each device holds full sequences for some headsAll-to-all before and after attentionLong contexts with many attention heads
Hybrid schemesCombine ring and head splittingMixed collectivesExtreme context lengths on large clusters
Main sequence parallelism approaches

The all-to-all head split, popularized by DeepSpeed Ulysses, is limited by the number of attention heads: the parallel degree cannot exceed it. Ring attention has no such limit but depends on overlapping transfers with computation, and that overlap becomes harder on slow interconnects. Frameworks increasingly use the label context parallelism for these attention-level methods and reserve sequence parallelism for the Megatron-style complement to tensor parallelism.

Sequence parallelism compared with other parallelism strategies

Sequence parallelism is one dimension of a multi-dimensional parallel layout and is almost never used alone. Each strategy splits a different axis of the training workload, and large runs combine them.

StrategyWhat is splitMain memory savingMain cost
Data parallelismBatch of examplesNone by itselfGradient all-reduce
FSDP or ZeROParameters, gradients and optimizer statesModel statesParameter gathering each step
Tensor parallelismWeight matrices inside layersWeights and some activationsFrequent collectives, needs fast links
Pipeline parallelismGroups of layersWeights per stagePipeline bubbles
Sequence parallelismTokens of each sequenceActivationsCollectives at block boundaries
Expert parallelismMixture-of-experts expertsExpert weightsAll-to-all token routing
What each parallelism strategy partitions

The key distinction is that sequence parallelism targets activation memory, which grows linearly with sequence length and, without memory-efficient kernels, attention scores grow quadratically. Weight-oriented strategies do nothing for that growth, which is why long-context training needs a token-splitting dimension.

Configuring sequence parallelism in practice

Configuring sequence parallelism starts with matching the parallel group to the network topology. Megatron-style sequence parallelism shares its group with tensor parallelism, so it normally stays within one node connected by NVLink or a similar high-bandwidth fabric.

SituationRecommended settingReason
Tensor parallelism already enabledTurn on Megatron-style sequence parallelismSaves activation memory at near-zero communication cost
Context of 32k tokens or moreAdd ring or all-to-all context parallelismAttention activations dominate memory
Slow inter-node networkKeep token splitting inside nodesBoundary collectives are latency sensitive
Short sequences, small modelSkip itOverhead outweighs the memory benefit
Decision guide for sequence parallelism

Sequence parallelism is usually combined with selective activation recomputation and memory-efficient attention kernels such as FlashAttention. Together they let a run fit longer sequences or larger micro-batches, which improves model FLOPs utilization because each kernel processes more work per launch.

Limitations and common pitfalls

The main limitation of sequence parallelism is that its savings apply only to activations. It does not reduce parameter or optimizer memory, so it must be paired with sharding strategies when the model itself is large.

  • Sequence length must divide evenly by the parallel degree, or padding wastes compute.
  • Causal masking creates load imbalance in ring attention unless tokens are reordered across devices.
  • Interconnect bandwidth caps scaling; crossing node boundaries often erases the gain.
  • Numerical differences can appear because reductions happen in a different order than in single-device runs.
  • Inference use is narrower: it helps long-prompt prefill but rarely token-by-token decoding.

In simple terms

Training a large language model means pushing very long pieces of text through it, and every word leaves behind intermediate results kept in memory. With long documents, those results no longer fit on one graphics card.

Sequence parallelism cuts each long text into slices and gives every card one slice to hold. It resembles a team proofreading a long report by splitting the pages: each person keeps only their own pages on the desk, and they compare notes only at fixed checkpoints. The result is that the team can handle much longer documents than any one person could alone.

A practical example

A research team is extending a 13-billion-parameter language model from a 4,096-token context to 32,768 tokens on nodes with eight GPUs of 80 GB each. With tensor parallelism of eight alone, a single 32k-token micro-batch runs out of memory because layer normalization and dropout activations are still replicated on every GPU.

The team first enables Megatron-style sequence parallelism within each node, which divides those activations by eight without adding communication volume. Attention memory is still too high, so they add ring-based context parallelism of two across node pairs and use FlashAttention with selective recomputation.

The run now fits a full 32k-token sequence per micro-batch. Measured throughput is only modestly below the original short-context configuration per token, and the team avoids the alternative of truncating training documents, which would have weakened the model’s long-range behavior.