Expert parallelism is a distributed computing strategy for mixture-of-experts (MoE) models in which the individual expert networks of each MoE layer are placed on different GPUs, and each token is sent over the network to whichever devices hold the experts its router selects. In AI infrastructure, it is the technique that makes models with hundreds of billions to trillions of parameters trainable and servable, because no single accelerator could hold all of their experts in memory.
Unlike strategies that split every computation evenly, expert parallelism exploits sparse activation: each token uses only a few experts, so the system moves tokens to experts rather than copying all weights everywhere. The price is heavy all-to-all communication and sensitivity to how evenly the router spreads its load.
How expert parallelism moves tokens to experts
An MoE layer replaces a dense feed-forward block with many expert sub-networks and a learned router. For each token, the router computes scores and selects the top-k experts, commonly one, two or eight out of dozens to several hundred. Under expert parallelism, every GPU in the expert-parallel group stores a subset of these experts.
A forward pass through the layer follows a fixed sequence. Each GPU runs the router on its local tokens, then a dispatch all-to-all sends every token to the GPUs owning its chosen experts. The experts process the tokens they receive, and a combine all-to-all returns the outputs to their original devices, where they are weighted by router scores and summed.
The following table summarizes the stages of one MoE layer and the dominant cost of each stage.
| Stage | What happens | Dominant cost |
|---|---|---|
| Routing | Router scores tokens and picks top-k experts | Small matrix multiply |
| Dispatch | Tokens sent to devices that own the experts | All-to-all network traffic |
| Expert compute | Each expert runs its feed-forward network | Matrix multiplies on local GPU |
| Combine | Expert outputs returned to source devices | Second all-to-all exchange |
| Aggregation | Outputs weighted by router scores and summed | Element-wise operations |
Because the router decides destinations at run time, traffic patterns change every batch. Communication libraries therefore use variable-size exchanges, and modern implementations overlap all-to-all transfers with expert computation to hide part of the latency.
Key parameters that shape performance
Expert parallelism performance depends on a small set of parameters that trade memory, compute balance and communication volume against one another. Choosing them well matters as much as raw hardware speed.
| Parameter | Typical range | Effect when increased |
|---|---|---|
| Expert-parallel degree | 8 to several hundred GPUs | Less memory per GPU, more network traffic |
| Experts per layer | 8 to 256 or more | More capacity, finer routing, smaller experts |
| Top-k per token | 1 to 8 | Better quality, proportionally more traffic |
| Capacity factor | About 1.0 to 1.5 | Fewer dropped tokens, more padding waste |
| Shared experts | 0 to 2 | Stable common features, extra dense compute |
The capacity factor sets how many tokens each expert accepts per batch relative to an even share. Tokens beyond capacity are dropped or rerouted, which harms quality; a generous factor wastes memory and compute on padding. Auxiliary load-balancing losses or bias-based balancing methods train the router to spread tokens evenly so that a modest capacity factor suffices.
Expert parallelism compared with other parallelism strategies
Expert parallelism is one dimension of a larger parallelism plan and is almost always combined with others. Each strategy partitions a different part of the workload and generates a different communication pattern.
| Strategy | What is split | Main collective | Best suited to |
|---|---|---|---|
| Data parallelism | Training batch | All-reduce of gradients | Any model that fits on one device |
| Tensor parallelism | Individual weight matrices | All-reduce within each layer | Very wide layers, fast intra-node links |
| Pipeline parallelism | Groups of layers | Point-to-point activations | Very deep models across nodes |
| Expert parallelism | Experts of MoE layers | All-to-all token exchange | Sparse MoE models with many experts |
| Sequence parallelism | Tokens of long sequences | All-gather and reduce-scatter | Long contexts and activation memory |
A common production layout applies expert parallelism to MoE layers while attention layers use data or tensor parallelism over the same GPUs. Frameworks map these groups onto the topology so that the bandwidth-hungry all-to-all runs over NVLink or similar high-speed fabric inside a node or rack wherever possible, with slower inter-node links reserved for lighter traffic.
Expert parallelism in training and inference
Expert parallelism serves both training and inference, but the bottlenecks differ. Training processes large batches, so expert computation is dense enough to amortize communication. Inference, especially the decode phase, processes few tokens per step, so each expert receives small batches and communication latency dominates.
| Aspect | Training | Inference |
|---|---|---|
| Tokens per step | Millions across the cluster | Tens to thousands per step |
| Main bottleneck | All-to-all bandwidth | All-to-all latency and small batches |
| Load imbalance impact | Slower steps and dropped tokens | Higher tail latency |
| Typical optimization | Balancing losses and overlap | Wide expert parallelism and batching |
Serving systems often use wide expert parallelism, spreading experts over dozens of GPUs so that every expert sees enough tokens from a large aggregate batch. This pairs naturally with disaggregated serving, where the prefill phase and decode phase run on separate pools sized for their different profiles. Redundant copies of popular experts further reduce hot spots when routing is skewed.
Limitations and common mistakes
The central weakness of expert parallelism is that network performance becomes a hard ceiling on model throughput. Teams that treat it as a simple memory-sharding technique often discover the limits only at scale.
- Ignoring topology: stretching an expert-parallel group across slow inter-node links can leave GPUs idle while waiting for all-to-all transfers.
- Underestimating routing skew: a few overloaded experts set the pace for the whole layer, so the slowest GPU determines step time.
- Setting capacity too low: aggressive token dropping silently degrades accuracy, especially on rare domains.
- Overlooking fault tolerance: losing one GPU removes experts that no other device holds unless replicas exist.
- Assuming lower total cost: MoE reduces compute per token, but memory for all experts and fast interconnects still cost significant capital.
The simple explanation
Some large AI models are built from many smaller specialists, called experts, and each word is handled by only a few of them. Expert parallelism puts different specialists on different computer chips, because all of them together are far too big for one chip.
Think of a hospital with many specialist departments in different buildings. Patients are not treated by every doctor; a receptionist sends each one to the right department and back. The system works only if the corridors are fast and no single department gets overwhelmed with patients. That is exactly the challenge of expert parallelism: fast connections and balanced workloads.
How it is used in practice
An AI lab serves an MoE language model with 256 experts per layer and eight active per token, far too large for a single eight-GPU server. Its engineers deploy the model across four servers with 32 GPUs, assigning eight experts to each GPU and running attention layers with data parallelism on the same devices.
Initial measurements show that a handful of experts handling code and mathematics receive three times the average traffic, raising tail latency. The team adds replicas of the busiest experts on underused GPUs and tunes batching so each decode step aggregates requests from hundreds of users. Throughput per GPU rises by roughly 40 percent, and the time per output token stays within the service target during peak load.