Query routing is the decision step that sends each incoming query to the most suitable destination, such as a specific vector index, a keyword search engine, a SQL database, an external tool or a particular language model, before retrieval or generation begins. In retrieval & knowledge systems for artificial intelligence, it acts as the dispatcher of a RAG pipeline: instead of searching every source for every question, the system first classifies the intent and then calls only the components that can answer it.
A well-designed router improves answer relevance, latency and cost at the same time, because irrelevant sources stop polluting the context window and expensive paths run only when they are needed. A poorly designed router becomes a single point of failure, since a wrong route can make a correct answer unreachable no matter how good the downstream retriever is.
How query routing works inside a retrieval pipeline
Query routing works by classifying the query against a set of named routes and then executing the route, or routes, that receive the highest score. Each route has a description, for example “product manuals for hardware support” or “monthly sales figures in the data warehouse”, and the router compares the incoming text with those descriptions.
The process usually has four stages. The system first normalizes the query, sometimes with query rewriting to resolve pronouns from the conversation history. It then scores every candidate route, applies a confidence threshold, and finally dispatches the query to one destination, several destinations in parallel, or a fallback route when no score is high enough.
The following table shows the typical stages and what each one decides.
| Stage | What it decides | Typical output |
|---|---|---|
| Normalization | Which standalone question is being asked | Rewritten query text |
| Route scoring | How well each route matches the intent | Score or probability per route |
| Thresholding | Whether any route is confident enough | Selected routes or none |
| Dispatch | Where the query is executed and how results merge | Retrieved passages or tool output |
Routing differs from retrieval itself. A retriever ranks documents inside one collection, while a router chooses which collection or capability is searched at all. In agentic RAG the same decision is often made by an agent with tool calling, but the underlying logic is identical.
Main types of query routers
Query routers differ mainly in how they compute the routing decision, and each technique trades accuracy against speed and maintenance effort. Production systems frequently combine two of them, with a cheap router in front and a more capable one as a fallback.
| Router type | Mechanism | Typical latency | Main weakness |
|---|---|---|---|
| Rule-based router | Keywords, regular expressions, metadata | Under 1 ms | Brittle with paraphrases |
| Semantic router | Embedding similarity to example utterances | 5-30 ms | Needs good example sets |
| Trained classifier | Fine-tuned small model on labeled queries | 5-50 ms | Requires labeled training data |
| LLM router | Language model picks a route from descriptions | 200-2,000 ms | Higher cost and variance |
| Agentic router | Agent calls tools iteratively | Several seconds | Hard to test and bound |
A semantic router embeds a handful of example utterances per route, often 10 to 50, and uses cosine similarity between the query embedding and those examples. It is fast and easy to extend, but it inherits the blind spots of the embedding model, for instance weak handling of negation or of numeric constraints.
An LLM router receives the route descriptions in its prompt and returns a structured choice, usually as JSON or a function call. It handles ambiguous and compound questions better than embeddings, but it adds a full model call to every request and must be constrained so that it cannot invent a route that does not exist.
Query routing compared with related retrieval techniques
Query routing is often confused with neighboring techniques because all of them operate on the query before or during retrieval. The distinction lies in what is being changed: the destination, the query text, or the ranking of results.
| Technique | What it changes | When it acts |
|---|---|---|
| Query routing | Destination of the query | Before retrieval |
| Query rewriting | Wording of the query | Before retrieval |
| Hybrid search | Combines dense and sparse scores | During retrieval |
| Metadata filtering | Subset of one index | During retrieval |
| Reranking | Order of retrieved candidates | After retrieval |
Metadata filtering and routing can overlap. A filter such as department = legal narrows one shared index, whereas routing selects entirely separate stores or tools. Teams often start with filters and move to routing once sources diverge in format, freshness requirements or access rules.
Designing and tuning a router in practice
A reliable router starts with clear, non-overlapping route definitions. When two routes describe similar content, scores become close and the router behaves unpredictably, so overlapping sources are better merged or separated by a sharper description.
The confidence threshold is the most important tuning parameter. A high threshold sends more queries to the fallback path and reduces wrong routes, while a low threshold answers more queries directly but increases misrouting. Teams usually set it on a labeled evaluation set of several hundred real queries and measure routing accuracy separately from final answer quality.
The decision table below summarizes common design choices.
| Situation | Recommended approach | Reason |
|---|---|---|
| Few stable sources | Rule-based or semantic router | Cheap, fast and predictable |
| Many overlapping intents | LLM router with structured output | Better at nuance and compound requests |
| High traffic, strict latency | Trained classifier | Low cost per query at scale |
| Uncertain or mixed queries | Multi-route dispatch with fusion | Avoids losing relevant sources |
| Sensitive data sources | Router plus access checks | Routing must not bypass permissions |
When a query legitimately needs several sources, the router can dispatch it to multiple routes in parallel and merge results with reciprocal rank fusion or a cross-encoder reranker. This multi-route pattern costs more but protects recall on questions that span domains.
Limitations and common failure modes
The central risk of query routing is that errors are silent: a misrouted query still returns fluent text, but it is grounded in the wrong source. Without logging of the chosen route, such failures look like retrieval or generation problems and are hard to diagnose.
- Route drift: new document types appear, but route descriptions and examples are never updated.
- Overconfident routing: the router always picks a route even when none fits, so the fallback never triggers.
- Conversation blindness: follow-up questions such as “and last year?” are routed without the earlier context.
- Latency stacking: an LLM router placed in front of an already slow pipeline doubles response time.
- Permission leakage: routing to a privileged tool without checking the user’s access rights.
Good practice is to log every routing decision with its score, review low-confidence cases regularly, and keep a general-purpose fallback route that searches broadly. Routing accuracy should be tracked as its own metric next to RAG evaluation scores.
The short version
Query routing is like the reception desk in a large hospital. You describe your problem once, and the receptionist sends you to the right department instead of making you visit every ward.
In an AI assistant, the router reads the question and decides where the answer most likely lives: a document library, a database, a calculator or a web search. When it chooses well, answers are faster, cheaper and more accurate. When it chooses badly, even a perfect search engine cannot help, because it is searching in the wrong place.
Where it makes a difference
Consider an internal assistant at a mid-sized manufacturing company with three separate knowledge sources: about 8,000 pages of equipment manuals in a vector index, a SQL warehouse with production and sales figures, and an HR policy library. In the first version, every question searched all three sources, and answers to numeric questions often quoted manuals instead of actual figures.
The team added a semantic router with 30 example questions per route and a confidence threshold tuned on 400 logged queries. Questions such as “how many units did line 2 produce in March” now go to a text-to-SQL tool, while maintenance questions go to the manuals index. Low-confidence questions are sent to all three sources and merged with reciprocal rank fusion.
After the change, routing accuracy reached roughly 94 percent on the evaluation set, average response time fell because most queries touched only one source, and the number of answers quoting the wrong source dropped sharply. The logged routing scores also showed which new topics needed a route of their own.