Gating network

Also known as: expert router

On this page 7

A gating network, also called an expert router, is the small learned component in a mixture of experts architecture that assigns each input to one or more expert subnetworks and sets how strongly each selected expert contributes to the output. In neural networks, it is the mechanism that makes conditional computation possible: the model holds many parameters but activates only a fraction of them for any given token or example.

Within Artificial intelligence research, gating networks became central to large language models because they allow parameter count to grow faster than compute cost. A sparse model with dozens of experts can match the quality of a much larger dense model while spending roughly the floating-point operations of a single expert per token.

How a gating network routes inputs

A gating network computes a score for every expert from the input representation and converts those scores into routing weights. In the most common form, the router is a single linear layer that multiplies the hidden state of a token by a learned router matrix, producing one logit per expert, followed by a softmax.

In sparse routing, only the top-k experts with the highest scores are kept, typically k equal to 1 or 2, and their weights are renormalized. The layer output is the weighted sum of the selected experts’ outputs. Because the gating weights are differentiable, gradients flow back into the router, so it learns which experts suit which inputs during ordinary training.

The table below summarizes the steps that happen inside a mixture of experts layer for each token.

StepOperationResult
ScoringLinear projection of the token stateOne logit per expert
NormalizationSoftmax or sigmoid over logitsRouting probabilities
SelectionKeep the top-k scoresA small active expert set
DispatchSend the token to chosen expertsExpert-specific computation
CombinationWeighted sum of expert outputsLayer output for the token
Routing steps inside a mixture of experts layer

Routing strategies and their parameters

Gating networks differ mainly in how many experts they activate and who chooses whom. The choice changes model quality, training stability and hardware efficiency, so it is one of the most consequential design decisions in a sparse model.

StrategyHow it selectsStrengthWeakness
Dense soft gatingAll experts, weighted by softmaxSmooth, stable trainingNo compute savings
Top-1 routingSingle best expert per tokenLowest compute and communicationHigher risk of imbalance
Top-2 routingTwo best experts per tokenBetter quality, more robustAbout twice the expert compute
Expert-choice routingEach expert picks its top tokensPerfect load balance by designSome tokens may be skipped
Noisy top-k gatingAdds tunable noise before selectionEncourages exploration of expertsExtra hyperparameters
Common gating strategies

Several numeric settings control the router. The capacity factor, usually between 1.0 and 1.25 in training, sets how many tokens each expert may accept per batch; tokens beyond that limit are dropped or passed through the residual connection. The number of experts commonly ranges from 8 to several hundred per layer, and some designs add shared experts that every token uses alongside the routed ones.

Keeping expert load balanced

The central difficulty of a gating network is routing collapse, a self-reinforcing loop in which a few experts receive most tokens, train faster, and therefore attract even more tokens. Unused experts stay undertrained, and the model loses the capacity it was designed to have.

Engineers counter this with auxiliary load-balancing losses that penalize uneven token distribution, a router z-loss that keeps router logits small for numerical stability, and bias-based balancing that adjusts per-expert offsets without adding a loss term. Each approach trades a little modeling freedom for stable, evenly used experts.

SymptomLikely causeTypical remedy
Few experts receive most tokensRouting collapseAuxiliary balancing loss or bias adjustment
Many tokens droppedCapacity factor too lowRaise capacity or use dropless kernels
Loss spikes during trainingLarge router logitsRouter z-loss, higher precision router
Experts learn redundant skillsWeak specialization pressureFine-grained experts plus shared experts
Balancing problems and typical remedies

A gating network is easily confused with other uses of the word gate in neural networks, yet its role is distinct. It selects which subnetwork runs, whereas most other gates control how much information passes within a single fixed computation.

MechanismWhat it controlsGranularity
Gating networkWhich experts process an inputPer token or per example
LSTM and GRU gatesFlow of memory through a recurrent cellPer hidden unit and time step
Attention mechanismWeighting of other tokens’ valuesPer token pair
HypernetworkGeneration of another network’s weightsPer task or input
Gating network versus neighboring concepts

The attention mechanism and the gating network both use softmax over learned scores, but attention mixes information between tokens, while the router mixes computation paths for one token. The mixture of experts term describes the whole architecture; the gating network is only its decision-making part.

Practical limitations and common mistakes

Sparse routing reduces arithmetic but not memory: all experts must be stored, so a model with a modest active parameter count can still require many accelerators to serve. Routing also introduces all-to-all communication when experts sit on different devices, which can dominate step time on slow interconnects.

  • Assuming experts are topic specialists: analyses often show that routers specialize on token types or syntax rather than human-readable subjects.
  • Ignoring inference batching: small batches leave many experts idle, so throughput gains seen in training may not appear in serving.
  • Fine-tuning the router carelessly: aggressive updates on narrow data can destabilize a well-balanced pretrained router; many teams freeze or regularize the router during adaptation.

In simple terms

A gating network works like a receptionist in a large clinic. Each patient arrives, the receptionist looks briefly at the problem, and sends the patient to the one or two specialists most likely to help. The other doctors stay free for other patients.

In an AI model, the patients are pieces of text and the specialists are small expert networks. Because only a few experts work on each piece, the model can be very large yet still fast. The difficult part is making sure the receptionist does not send everyone to the same doctor.

A practical example

A research team trains a language model with 64 experts per layer and top-2 routing, so each token activates only about one thirty-second of the expert parameters. Early in training, monitoring dashboards show that four experts receive nearly half of all tokens while a dozen experts receive almost none, and the training loss starts to plateau.

The team adds an auxiliary load-balancing loss with a small coefficient, introduces a router z-loss, and raises the capacity factor from 1.0 to 1.25. Within a few thousand steps the token share per expert settles close to the ideal average, dropped tokens fall below one percent, and validation loss resumes its decline. The final sparse model reaches quality comparable to a dense model with several times more active compute per token.