All-reduce (also written allreduce) is a collective communication operation in which every participating process contributes a buffer of values, the buffers are combined with a reduction function such as sum or average, and every process receives the same combined result. In AI infrastructure, all-reduce is the operation that keeps model replicas synchronized during distributed training, most visibly when gradients computed on many GPUs are summed before each optimizer step.
Because it runs at least once per training step and moves data proportional to the model size, all-reduce is often the dominant communication cost of data parallelism. Its efficiency depends on the algorithm, the network topology and the overlap with computation, which is why it receives so much attention in cluster design.
How all-reduce combines and redistributes data
All-reduce is logically equivalent to a reduce followed by a broadcast: values are combined into one result, which is then sent to all participants. Efficient implementations do not perform these phases naively through a single root, because that root would become a bandwidth bottleneck. Instead, they decompose the operation into a reduce-scatter, where each process ends up with a fully reduced slice of the buffer, followed by an all-gather, where the slices are exchanged until every process holds the complete result.
The reduction function must be associative, and usually commutative, so that partial results can be combined in any order. Sum is by far the most common choice for gradients; maximum, minimum and product are also supported by standard libraries. Floating-point addition is not perfectly associative, so different algorithms can produce bitwise different results, a detail that matters for reproducibility.
The table below places all-reduce among the related collective operations defined by the MPI standard and implemented in GPU libraries.
| Collective | Input per process | Output per process |
|---|---|---|
| All-reduce | Full buffer | Full reduced buffer |
| Reduce | Full buffer | Reduced buffer on one root only |
| Reduce-scatter | Full buffer | One reduced slice |
| All-gather | One slice | All slices concatenated |
| Broadcast | Buffer on root only | Copy of root buffer |
Ring, tree and hierarchical algorithms
Several algorithms implement all-reduce, and the best one depends on message size, process count and topology. In the ring algorithm, processes form a logical ring and pass chunks to their neighbor in 2(N−1) steps. Each process sends and receives about 2(N−1)/N times the buffer size, which approaches twice the buffer size and is bandwidth-optimal, but latency grows linearly with the number of processes.
Tree algorithms, including the double binary tree used in NCCL, reduce latency to a logarithmic number of steps and suit small messages or very large clusters. Hierarchical all-reduce first reduces inside a node over fast links such as NVLink, then across nodes over InfiniBand or Ethernet, and finally distributes the result inside each node. Some network switches can also perform in-network reduction, which offloads the arithmetic from the GPUs.
| Algorithm | Latency steps | Bandwidth behavior | Best fit |
|---|---|---|---|
| Ring | Linear in N | Bandwidth-optimal | Large gradients, moderate N |
| Double binary tree | Logarithmic in N | Near-optimal | Small messages, large N |
| Recursive halving-doubling | Logarithmic in N | Good for powers of two | CPU and MPI clusters |
| Hierarchical | Intra-node plus inter-node | Exploits fast local links | Multi-GPU nodes |
| In-network reduction | Very low | Reduces traffic on links | Switches with reduction support |
All-reduce in distributed training
In data parallelism, each GPU holds a full model replica, processes a different mini-batch and computes local gradients. An all-reduce then averages the gradients across all replicas, so every GPU applies an identical update and the replicas never diverge. Frameworks such as PyTorch DistributedDataParallel group gradients into buckets of tens of megabytes and launch the all-reduce for each bucket as soon as its gradients are ready, overlapping communication with the rest of the backward pass.
Other parallelism strategies use all-reduce differently. Tensor parallelism issues all-reduce operations inside each layer to combine partial matrix products, which makes it latency-sensitive and usually confines it to GPUs within one node. Sharded approaches such as FSDP and ZeRO replace the single gradient all-reduce with a reduce-scatter and an all-gather, which moves a similar volume of data while storing only a shard of the parameters, gradients and optimizer state on each GPU.
| Strategy | What is reduced | Frequency | Sensitivity |
|---|---|---|---|
| Data parallelism | Gradients | Once per step, bucketed | Bandwidth |
| Tensor parallelism | Partial activations | Several times per layer | Latency |
| FSDP and ZeRO | Gradients via reduce-scatter | Per layer or unit | Bandwidth and memory |
| Expert parallelism | Mostly all-to-all instead | Per mixture-of-experts layer | Network topology |
Performance factors and tuning
All-reduce performance is usually reported as bus bandwidth, a normalized figure that can be compared with the hardware link speed. A well-tuned job reaches a large fraction of the link bandwidth for large messages, while small messages are dominated by latency rather than throughput.
- Interconnect speed and topology set the upper bound; intra-node links are typically an order of magnitude faster than inter-node networks.
- Bucket size balances overlap with per-call overhead; very small buckets waste latency, very large ones delay the start of communication.
- Gradient compression in lower precision such as BF16 or FP8 halves or quarters the traffic, at some risk to numerical accuracy.
- Straggler processes slow every step, because all-reduce is a synchronization point and finishes only when the slowest participant finishes.
Failure modes and common misconceptions
Because all-reduce is blocking for all participants, a single faulty GPU, link or process can stall an entire cluster. A mismatch in the order or shape of collective calls across ranks produces a hang rather than an error, which is one of the hardest problems to diagnose in distributed training. Timeouts and watchdogs in communication libraries exist precisely to surface these cases.
A common misconception is that adding GPUs always shortens training proportionally. In reality, communication overhead grows in relative terms as the per-GPU batch shrinks, so scaling efficiency depends on keeping computation large enough to hide all-reduce time. Another misconception is that all-reduce averages automatically; libraries typically compute a sum, and the division by the number of replicas is performed by the framework or must be added explicitly.
Explained without jargon
Imagine eight students each grading a different pile of the same exam and keeping a tally of common mistakes. At the end, every student needs the class-wide total. Instead of sending all tallies to one teacher, they pass partial sums around the room until everyone holds the same final total.
All-reduce does this for computers. Each GPU computes its own numbers, the numbers are added together, and every GPU gets the same answer. Large AI models are trained this way so that many machines can learn from different data while staying perfectly in step.
A typical use case
A research team fine-tunes a language model with about seven billion parameters on a cluster of four servers, each with eight GPUs connected by fast in-node links and a 400 Gb/s network between servers. They use data parallelism with BF16 gradients, so each training step must synchronize roughly 14 GB of gradient data across 32 GPUs.
Profiling shows that the first configuration spends almost a third of each step waiting for communication. The engineers enable hierarchical all-reduce, increase the gradient bucket size so that communication starts earlier and overlaps with the backward pass, and raise the per-GPU batch size. After these changes, most of the all-reduce time is hidden behind computation, and step time drops noticeably without any change to the model or its accuracy.