Sequence parallelism is a distributed training technique that partitions activations along the sequence (token) dimension across several accelerators, so that each device stores and processes only a slice of every input sequence. In AI infrastructure it is used mainly to reduce activation memory in large transformer models, which makes longer context windows and larger micro-batches possible on the same hardware.
The term has two closely related uses. In the narrow Megatron-LM sense, it complements tensor parallelism by splitting the layers that tensor parallelism leaves replicated. In a broader sense, it covers any scheme that shards tokens, including attention-level approaches that are now often called context parallelism.
How sequence parallelism splits transformer activations
Sequence parallelism works by giving each device a contiguous chunk of tokens for the operations that act on tokens independently. Layer normalization, dropout and residual additions treat every position separately, so they can run on a shard of the sequence without any communication.
In the Megatron-LM design, tensor parallelism already splits the attention and MLP weight matrices across a group of GPUs, but the activations between those blocks remain fully replicated on every device. Sequence parallelism shards exactly those replicated regions. The usual all-reduce after a tensor-parallel block is replaced by a reduce-scatter, which leaves each device with its own token slice, and an all-gather restores the full sequence before the next tensor-parallel block.
Because one all-reduce is mathematically equivalent to a reduce-scatter followed by an all-gather, this form of sequence parallelism adds almost no extra communication volume. Its benefit is that activation memory for the non-tensor-parallel regions drops by a factor equal to the tensor-parallel group size.
The table below summarizes what happens to each part of a transformer layer.
| Layer component | Without sequence parallelism | With sequence parallelism |
|---|---|---|
| Layer normalization | Full sequence replicated on each GPU | Each GPU normalizes its own token slice |
| Dropout and residuals | Activations replicated | Activations sharded by tokens |
| Attention and MLP weights | Split by tensor parallelism | Unchanged, still split by tensor parallelism |
| Block boundary communication | All-reduce | Reduce-scatter plus all-gather |
Variants of sequence parallelism
Several variants of sequence parallelism exist, and they differ mainly in how attention is handled, since attention is the one operation where every token must see other tokens. The choice determines how far the technique scales with context length.
| Approach | Attention handling | Main communication | Typical use |
|---|---|---|---|
| Megatron-style | Attention stays under tensor parallelism | Reduce-scatter and all-gather | Paired with tensor parallelism inside a node |
| Ring attention | Key and value blocks rotate around a ring of devices | Point-to-point transfers overlapped with compute | Very long contexts across many devices |
| All-to-all head split | Tokens regrouped so each device holds full sequences for some heads | All-to-all before and after attention | Long contexts with many attention heads |
| Hybrid schemes | Combine ring and head splitting | Mixed collectives | Extreme context lengths on large clusters |
The all-to-all head split, popularized by DeepSpeed Ulysses, is limited by the number of attention heads: the parallel degree cannot exceed it. Ring attention has no such limit but depends on overlapping transfers with computation, and that overlap becomes harder on slow interconnects. Frameworks increasingly use the label context parallelism for these attention-level methods and reserve sequence parallelism for the Megatron-style complement to tensor parallelism.
Sequence parallelism compared with other parallelism strategies
Sequence parallelism is one dimension of a multi-dimensional parallel layout and is almost never used alone. Each strategy splits a different axis of the training workload, and large runs combine them.
| Strategy | What is split | Main memory saving | Main cost |
|---|---|---|---|
| Data parallelism | Batch of examples | None by itself | Gradient all-reduce |
| FSDP or ZeRO | Parameters, gradients and optimizer states | Model states | Parameter gathering each step |
| Tensor parallelism | Weight matrices inside layers | Weights and some activations | Frequent collectives, needs fast links |
| Pipeline parallelism | Groups of layers | Weights per stage | Pipeline bubbles |
| Sequence parallelism | Tokens of each sequence | Activations | Collectives at block boundaries |
| Expert parallelism | Mixture-of-experts experts | Expert weights | All-to-all token routing |
The key distinction is that sequence parallelism targets activation memory, which grows linearly with sequence length and, without memory-efficient kernels, attention scores grow quadratically. Weight-oriented strategies do nothing for that growth, which is why long-context training needs a token-splitting dimension.
Configuring sequence parallelism in practice
Configuring sequence parallelism starts with matching the parallel group to the network topology. Megatron-style sequence parallelism shares its group with tensor parallelism, so it normally stays within one node connected by NVLink or a similar high-bandwidth fabric.
| Situation | Recommended setting | Reason |
|---|---|---|
| Tensor parallelism already enabled | Turn on Megatron-style sequence parallelism | Saves activation memory at near-zero communication cost |
| Context of 32k tokens or more | Add ring or all-to-all context parallelism | Attention activations dominate memory |
| Slow inter-node network | Keep token splitting inside nodes | Boundary collectives are latency sensitive |
| Short sequences, small model | Skip it | Overhead outweighs the memory benefit |
Sequence parallelism is usually combined with selective activation recomputation and memory-efficient attention kernels such as FlashAttention. Together they let a run fit longer sequences or larger micro-batches, which improves model FLOPs utilization because each kernel processes more work per launch.
Limitations and common pitfalls
The main limitation of sequence parallelism is that its savings apply only to activations. It does not reduce parameter or optimizer memory, so it must be paired with sharding strategies when the model itself is large.
- Sequence length must divide evenly by the parallel degree, or padding wastes compute.
- Causal masking creates load imbalance in ring attention unless tokens are reordered across devices.
- Interconnect bandwidth caps scaling; crossing node boundaries often erases the gain.
- Numerical differences can appear because reductions happen in a different order than in single-device runs.
- Inference use is narrower: it helps long-prompt prefill but rarely token-by-token decoding.
In simple terms
Training a large language model means pushing very long pieces of text through it, and every word leaves behind intermediate results kept in memory. With long documents, those results no longer fit on one graphics card.
Sequence parallelism cuts each long text into slices and gives every card one slice to hold. It resembles a team proofreading a long report by splitting the pages: each person keeps only their own pages on the desk, and they compare notes only at fixed checkpoints. The result is that the team can handle much longer documents than any one person could alone.
A practical example
A research team is extending a 13-billion-parameter language model from a 4,096-token context to 32,768 tokens on nodes with eight GPUs of 80 GB each. With tensor parallelism of eight alone, a single 32k-token micro-batch runs out of memory because layer normalization and dropout activations are still replicated on every GPU.
The team first enables Megatron-style sequence parallelism within each node, which divides those activations by eight without adding communication volume. Attention memory is still too high, so they add ring-based context parallelism of two across node pairs and use FlashAttention with selective recomputation.
The run now fits a full 32k-token sequence per micro-batch. Measured throughput is only modestly below the original short-context configuration per token, and the team avoids the alternative of truncating training documents, which would have weakened the model’s long-range behavior.