Model routing is the practice of directing each incoming request to one of several AI models based on rules or a learned predictor, so that every query is served by the model that offers the best balance of quality, latency and cost for that particular task. In AI infrastructure, a router sits in front of the model serving layer and decides, per request, whether a small fast model, a large reasoning model or a specialized model should answer.
The motivation is economic and technical. Prices and response times between small and frontier language models often differ by one to two orders of magnitude, yet many production queries are simple enough for the cheaper model. Routing captures those savings while reserving expensive capacity for the requests that truly need it, which makes it a core cost-control technique in artificial intelligence deployments.
How a model router makes decisions
A router receives the request, extracts features such as prompt length, detected language, task type, user tier or the output of a small classifier, and maps them to a target model. The decision happens before generation, so it must be fast, typically a few milliseconds, or it erodes the latency savings it is meant to create.
Routers differ mainly in when and how they decide. Some choose once and commit; others try a cheap model first and escalate only when the answer looks unreliable. The table below compares the main strategies.
| Strategy | How it decides | Added latency | Typical use |
|---|---|---|---|
| Rule-based routing | Fixed rules on task, user tier or length | Negligible | Clear product segments |
| Classifier routing | Small model predicts difficulty or category | Milliseconds | Mixed traffic with varied difficulty |
| Embedding similarity routing | Nearest labeled examples choose the model | Milliseconds | Many task types, few labels |
| Cascade with escalation | Cheap model first, larger model on low confidence | Extra call when escalated | Cost-sensitive workloads |
| Learned preference router | Trained on outcomes or human preferences | Milliseconds | Large, stable traffic |
Signals and parameters that drive routing
A router is only as good as its signals. The most useful ones are cheap to compute and correlate with the difference in quality between candidate models, not merely with the topic of the request.
| Signal | What it indicates | Typical routing effect |
|---|---|---|
| Task category | Classification, extraction, coding or reasoning | Reasoning and code go to stronger models |
| Prompt and context length | Need for a long context window | Long inputs go to long-context models |
| Predicted difficulty score | Chance the small model fails | Above threshold goes to large model |
| Self-reported confidence | Uncertainty in the cheap model’s answer | Low confidence triggers escalation |
| Language or modality | Non-English text, images or audio | Specialized or multimodal model |
| Service tier and budget | Customer plan or remaining spend | Premium tiers get stronger models |
The central tuning parameter is the routing threshold: the difficulty or confidence level at which a request moves to the stronger model. Lowering it raises quality and cost together. Teams usually plot a cost-quality curve on a labeled set and pick the threshold where further spending yields little improvement.
Model routing compared with related infrastructure
Model routing is often confused with neighboring infrastructure functions. Load balancing spreads identical requests across replicas of the same model, while routing chooses between different models. Gateways frequently combine both, which blurs the terms.
| Concept | What it decides | Relation to model routing |
|---|---|---|
| Model serving | How a model is hosted and exposed | Routing selects among served models |
| Inference server | How requests are batched and executed on hardware | Receives the routed request |
| Load balancing | Which replica of one model handles a call | Operates after the model is chosen |
| Prompt caching | Whether a repeated prefix can be reused | Routing can favor the model with a warm cache |
| Fallback | What to do when a provider fails | A routing rule triggered by errors |
| Mixture of experts | Which expert layers process a token | Routing inside one model, not between models |
The comparison with mixture of experts is especially common. Both use a gating decision, but a mixture-of-experts gate is learned jointly with the model and works per token inside a single network, whereas a model router works per request across separate models, often from different providers.
Designing and operating a router
A production router needs the same discipline as any other critical service: evaluation data, monitoring and safe fallbacks. The following decisions shape most implementations.
| Decision | Recommended practice | Reason |
|---|---|---|
| Evaluation set | Representative prompts graded on every candidate model | Shows where the small model is enough |
| Default route | Send uncertain cases to the stronger model | Protects quality when the router is unsure |
| Fallback | Secondary model or provider on errors and timeouts | Keeps the service available |
| Monitoring | Track route share, cost, latency and quality per route | Reveals drift in traffic or models |
| Consistency | Pin a conversation to one model where possible | Avoids style and memory jumps mid-dialogue |
| Re-evaluation | Retest routes when a model version changes | Thresholds tuned on old models go stale |
Routing interacts with GPU inference capacity as well. Shifting most traffic to a small self-hosted model can saturate its hardware at peak times, so routers often consider queue depth and current load alongside request difficulty.
Risks and common mistakes
- Optimizing cost without measuring quality. Savings look excellent until users notice weaker answers on hard queries.
- Relying on model self-confidence. Language models are often confidently wrong, so confidence alone is a weak escalation signal.
- Slow routers. A router that calls a large model to decide wipes out the latency benefit.
- Hidden behavior differences. Models differ in refusal policy, formatting and tool-calling conventions.
- Stale thresholds. A new model version silently changes the cost-quality curve.
Compliance adds another constraint. When models run in different regions or under different providers, the router must enforce data residency and contractual restrictions, so that sensitive requests are never sent to a model that is not approved for that data.
Put simply
Think of a hospital reception desk. A person with a simple question is helped by the nurse at the front, while a complicated case is sent straight to a specialist. Nobody would send every visitor to the most senior surgeon.
Model routing applies the same idea to AI models. A router looks at each question and sends easy ones to a quick, inexpensive model and hard ones to a more capable and more expensive model. Users get good answers, and the organization pays far less than if every question went to the biggest model.
In practice
A software company runs an in-app assistant that handles about 2 million requests per month. At first, every request goes to a large frontier model, and the monthly inference bill becomes the largest line in the product’s operating costs.
Engineers grade 3,000 real requests on both the large model and a small model and find that most simple requests, such as reformatting text or answering documentation lookups, are handled equally well by the small model. They train a lightweight classifier that predicts when the small model is likely to fail and route only those requests upward. After launch, roughly two thirds of traffic stays on the small model, median latency falls, and monthly inference costs drop substantially. User satisfaction scores stay flat, and a weekly report on quality per route lets the team adjust the threshold whenever either model is updated.