Expert parallelism

On this page 7

Expert parallelism is a distributed computing strategy for mixture-of-experts (MoE) models in which the individual expert networks of each MoE layer are placed on different GPUs, and each token is sent over the network to whichever devices hold the experts its router selects. In AI infrastructure, it is the technique that makes models with hundreds of billions to trillions of parameters trainable and servable, because no single accelerator could hold all of their experts in memory.

Unlike strategies that split every computation evenly, expert parallelism exploits sparse activation: each token uses only a few experts, so the system moves tokens to experts rather than copying all weights everywhere. The price is heavy all-to-all communication and sensitivity to how evenly the router spreads its load.

How expert parallelism moves tokens to experts

An MoE layer replaces a dense feed-forward block with many expert sub-networks and a learned router. For each token, the router computes scores and selects the top-k experts, commonly one, two or eight out of dozens to several hundred. Under expert parallelism, every GPU in the expert-parallel group stores a subset of these experts.

A forward pass through the layer follows a fixed sequence. Each GPU runs the router on its local tokens, then a dispatch all-to-all sends every token to the GPUs owning its chosen experts. The experts process the tokens they receive, and a combine all-to-all returns the outputs to their original devices, where they are weighted by router scores and summed.

The following table summarizes the stages of one MoE layer and the dominant cost of each stage.

StageWhat happensDominant cost
RoutingRouter scores tokens and picks top-k expertsSmall matrix multiply
DispatchTokens sent to devices that own the expertsAll-to-all network traffic
Expert computeEach expert runs its feed-forward networkMatrix multiplies on local GPU
CombineExpert outputs returned to source devicesSecond all-to-all exchange
AggregationOutputs weighted by router scores and summedElement-wise operations
Stages of an expert-parallel MoE layer

Because the router decides destinations at run time, traffic patterns change every batch. Communication libraries therefore use variable-size exchanges, and modern implementations overlap all-to-all transfers with expert computation to hide part of the latency.

Key parameters that shape performance

Expert parallelism performance depends on a small set of parameters that trade memory, compute balance and communication volume against one another. Choosing them well matters as much as raw hardware speed.

ParameterTypical rangeEffect when increased
Expert-parallel degree8 to several hundred GPUsLess memory per GPU, more network traffic
Experts per layer8 to 256 or moreMore capacity, finer routing, smaller experts
Top-k per token1 to 8Better quality, proportionally more traffic
Capacity factorAbout 1.0 to 1.5Fewer dropped tokens, more padding waste
Shared experts0 to 2Stable common features, extra dense compute
Parameters and their effects

The capacity factor sets how many tokens each expert accepts per batch relative to an even share. Tokens beyond capacity are dropped or rerouted, which harms quality; a generous factor wastes memory and compute on padding. Auxiliary load-balancing losses or bias-based balancing methods train the router to spread tokens evenly so that a modest capacity factor suffices.

Expert parallelism compared with other parallelism strategies

Expert parallelism is one dimension of a larger parallelism plan and is almost always combined with others. Each strategy partitions a different part of the workload and generates a different communication pattern.

StrategyWhat is splitMain collectiveBest suited to
Data parallelismTraining batchAll-reduce of gradientsAny model that fits on one device
Tensor parallelismIndividual weight matricesAll-reduce within each layerVery wide layers, fast intra-node links
Pipeline parallelismGroups of layersPoint-to-point activationsVery deep models across nodes
Expert parallelismExperts of MoE layersAll-to-all token exchangeSparse MoE models with many experts
Sequence parallelismTokens of long sequencesAll-gather and reduce-scatterLong contexts and activation memory
Comparison of parallelism strategies

A common production layout applies expert parallelism to MoE layers while attention layers use data or tensor parallelism over the same GPUs. Frameworks map these groups onto the topology so that the bandwidth-hungry all-to-all runs over NVLink or similar high-speed fabric inside a node or rack wherever possible, with slower inter-node links reserved for lighter traffic.

Expert parallelism in training and inference

Expert parallelism serves both training and inference, but the bottlenecks differ. Training processes large batches, so expert computation is dense enough to amortize communication. Inference, especially the decode phase, processes few tokens per step, so each expert receives small batches and communication latency dominates.

AspectTrainingInference
Tokens per stepMillions across the clusterTens to thousands per step
Main bottleneckAll-to-all bandwidthAll-to-all latency and small batches
Load imbalance impactSlower steps and dropped tokensHigher tail latency
Typical optimizationBalancing losses and overlapWide expert parallelism and batching
Training versus inference considerations

Serving systems often use wide expert parallelism, spreading experts over dozens of GPUs so that every expert sees enough tokens from a large aggregate batch. This pairs naturally with disaggregated serving, where the prefill phase and decode phase run on separate pools sized for their different profiles. Redundant copies of popular experts further reduce hot spots when routing is skewed.

Limitations and common mistakes

The central weakness of expert parallelism is that network performance becomes a hard ceiling on model throughput. Teams that treat it as a simple memory-sharding technique often discover the limits only at scale.

  • Ignoring topology: stretching an expert-parallel group across slow inter-node links can leave GPUs idle while waiting for all-to-all transfers.
  • Underestimating routing skew: a few overloaded experts set the pace for the whole layer, so the slowest GPU determines step time.
  • Setting capacity too low: aggressive token dropping silently degrades accuracy, especially on rare domains.
  • Overlooking fault tolerance: losing one GPU removes experts that no other device holds unless replicas exist.
  • Assuming lower total cost: MoE reduces compute per token, but memory for all experts and fast interconnects still cost significant capital.

The simple explanation

Some large AI models are built from many smaller specialists, called experts, and each word is handled by only a few of them. Expert parallelism puts different specialists on different computer chips, because all of them together are far too big for one chip.

Think of a hospital with many specialist departments in different buildings. Patients are not treated by every doctor; a receptionist sends each one to the right department and back. The system works only if the corridors are fast and no single department gets overwhelmed with patients. That is exactly the challenge of expert parallelism: fast connections and balanced workloads.

How it is used in practice

An AI lab serves an MoE language model with 256 experts per layer and eight active per token, far too large for a single eight-GPU server. Its engineers deploy the model across four servers with 32 GPUs, assigning eight experts to each GPU and running attention layers with data parallelism on the same devices.

Initial measurements show that a handful of experts handling code and mathematics receive three times the average traffic, raising tail latency. The team adds replicas of the busiest experts on underused GPUs and tunes batching so each decode step aggregates requests from hundreds of users. Throughput per GPU rises by roughly 40 percent, and the time per output token stays within the service target during peak load.