Mixture of experts (MoE) is a neural network architecture in which a learned router, or gating network, selects a small subset of specialized subnetworks, called experts, to process each input. Only the chosen experts are computed, so the model can hold far more parameters than it uses for any single token. In neural networks for language, vision and speech, MoE is the main technique for growing model capacity without a proportional increase in compute per prediction.
Modern MoE models usually replace the feed-forward block of a transformer layer with a set of expert feed-forward networks, while attention layers stay shared. This design is called sparse activation or conditional computation, in contrast to dense models in which every parameter participates in every forward pass.
How routing and experts work
An MoE layer works in three steps: score, select and combine. For each token, the router, typically a single linear layer followed by softmax, produces a score for every expert. The top-k experts with the highest scores receive the token, each computes its output, and the layer returns a weighted sum of those outputs using the router scores as weights.
The value of k is the central design choice. Top-1 routing minimizes compute and communication; top-2 routing is a common compromise that gives smoother training and some redundancy. Models with many small, fine-grained experts sometimes activate more of them per token and add one or more shared experts that every token always uses for common knowledge.
The following table lists the parameters that define an MoE layer and their typical ranges.
| Parameter | Typical range | What it controls |
|---|---|---|
| Number of experts | 8 to several hundred per layer | Total capacity and memory footprint |
| Experts per token (k) | 1 to 8 | Compute per token and output quality |
| Capacity factor | About 1.0 to 2.0 | Buffer for uneven expert load |
| Shared experts | 0 to 2 | Knowledge used by every token |
| MoE layer frequency | Every layer or every second layer | Balance of sparse and dense compute |
Load balancing and training stability
Load balancing is the main engineering problem of MoE, because an untrained router tends to favor a few experts that then improve faster and attract even more tokens. Left unchecked, this feedback loop produces expert collapse, where most experts sit idle and the model behaves like a much smaller dense network.
Each expert also has a fixed capacity, the number of tokens it can accept per batch. Tokens routed to a full expert are dropped and pass through the layer only via the residual connection, which hurts quality. Several countermeasures exist, each with costs.
| Technique | Mechanism | Trade-off |
|---|---|---|
| Auxiliary balancing loss | Penalizes uneven token distribution | Can conflict with the main objective |
| Router z-loss | Keeps router logits small and stable | Adds a tuning coefficient |
| Noise in router scores | Encourages exploration of experts early | Slightly less deterministic training |
| Bias-based balancing | Adjusts per-expert bias without extra loss | Requires careful update rules |
| Expert-choice routing | Experts pick tokens instead of the reverse | Some tokens may get no expert |
MoE models are also more sensitive to numerical precision in the router. Computing router probabilities in higher precision is a common safeguard against training instabilities that appear as sudden loss spikes.
Mixture of experts compared with dense models
Compared with a dense model of equal compute per token, an MoE model usually reaches better quality for the same training budget, because it can store more knowledge in its additional parameters. The cost moves from arithmetic to memory and communication: all experts must be stored somewhere, even though only a few run for each token.
| Property | Dense model | MoE model |
|---|---|---|
| Parameters used per token | All parameters | Router plus selected experts |
| Training compute for given quality | Higher | Often substantially lower |
| Memory to hold weights | Proportional to active size | Proportional to total size |
| Serving complexity | Straightforward | Expert parallelism and routing overhead |
| Fine-tuning behavior | Well understood | More prone to overfitting on small data |
A model is therefore described by two numbers: total parameters and active parameters. A model with roughly 47 billion total and about 13 billion active parameters, a configuration seen in openly released eight-expert models with top-2 routing, computes like a mid-sized dense model but must load all its weights into memory.
Serving and deployment considerations
Serving an MoE model efficiently requires expert parallelism, which places different experts on different accelerators and exchanges tokens between devices with all-to-all communication. Interconnect bandwidth therefore matters as much as raw compute, and small batches make poor use of experts because each one receives only a few tokens.
- Memory is the binding constraint: total parameters decide how many accelerators are needed, regardless of active size.
- Batching improves utilization: larger batches spread tokens evenly across experts.
- Quantization helps more than usual: compressing rarely used experts reduces memory with little quality loss.
- Expert offloading to host memory is possible for local use, but it adds latency for each expert swap.
Experts rarely specialize in human-readable topics. Analyses commonly find that routing follows token types, syntax and positions more than subjects such as biology or law, so the word expert should be read as a technical term rather than a promise of domain specialization.
Origins and variants
The mixture of experts idea predates deep learning; it was proposed in the early 1990s as an ensemble with a learned gate that weighted several small networks. The modern sparse form appeared with sparsely gated MoE layers in recurrent networks and later in transformers, where top-1 and top-2 routing made trillion-parameter-scale models trainable.
Variants now include soft MoE, which mixes token representations instead of routing discrete tokens, and MoE applied to attention heads or to vision transformers. All share the same principle: conditional computation that decouples capacity from cost.
Stripped to the essentials
A mixture of experts works like a large clinic with many specialists and a receptionist who decides whom each patient sees. Instead of every doctor examining every patient, each visitor is sent to the two or three who fit best. The clinic employs many experts, yet each visit takes only a little of their combined time.
The benefit is a much bigger pool of knowledge at a similar cost per answer. The drawback is that the whole staff still needs office space, which in a computer means memory.
Where it is applied
A software company wants an in-house assistant for code review and documentation that runs on its own servers. A dense model with the needed quality would be too slow on its hardware, so the team chooses an openly available MoE model with eight experts and top-2 routing per layer.
Generation speed is close to that of a dense model roughly one quarter of its total size, because only two experts run for each token. The team still needs enough accelerator memory for all experts, so it applies 8-bit quantization and spreads the experts across two accelerators. The result is interactive response times for dozens of developers, with output quality that the smaller dense alternatives did not reach in their internal evaluation.