Query routing

On this page 7

Query routing is the decision step that sends each incoming query to the most suitable destination, such as a specific vector index, a keyword search engine, a SQL database, an external tool or a particular language model, before retrieval or generation begins. In retrieval & knowledge systems for artificial intelligence, it acts as the dispatcher of a RAG pipeline: instead of searching every source for every question, the system first classifies the intent and then calls only the components that can answer it.

A well-designed router improves answer relevance, latency and cost at the same time, because irrelevant sources stop polluting the context window and expensive paths run only when they are needed. A poorly designed router becomes a single point of failure, since a wrong route can make a correct answer unreachable no matter how good the downstream retriever is.

How query routing works inside a retrieval pipeline

Query routing works by classifying the query against a set of named routes and then executing the route, or routes, that receive the highest score. Each route has a description, for example “product manuals for hardware support” or “monthly sales figures in the data warehouse”, and the router compares the incoming text with those descriptions.

The process usually has four stages. The system first normalizes the query, sometimes with query rewriting to resolve pronouns from the conversation history. It then scores every candidate route, applies a confidence threshold, and finally dispatches the query to one destination, several destinations in parallel, or a fallback route when no score is high enough.

The following table shows the typical stages and what each one decides.

StageWhat it decidesTypical output
NormalizationWhich standalone question is being askedRewritten query text
Route scoringHow well each route matches the intentScore or probability per route
ThresholdingWhether any route is confident enoughSelected routes or none
DispatchWhere the query is executed and how results mergeRetrieved passages or tool output
Stages of a query routing step

Routing differs from retrieval itself. A retriever ranks documents inside one collection, while a router chooses which collection or capability is searched at all. In agentic RAG the same decision is often made by an agent with tool calling, but the underlying logic is identical.

Main types of query routers

Query routers differ mainly in how they compute the routing decision, and each technique trades accuracy against speed and maintenance effort. Production systems frequently combine two of them, with a cheap router in front and a more capable one as a fallback.

Router typeMechanismTypical latencyMain weakness
Rule-based routerKeywords, regular expressions, metadataUnder 1 msBrittle with paraphrases
Semantic routerEmbedding similarity to example utterances5-30 msNeeds good example sets
Trained classifierFine-tuned small model on labeled queries5-50 msRequires labeled training data
LLM routerLanguage model picks a route from descriptions200-2,000 msHigher cost and variance
Agentic routerAgent calls tools iterativelySeveral secondsHard to test and bound
Common router implementations compared

A semantic router embeds a handful of example utterances per route, often 10 to 50, and uses cosine similarity between the query embedding and those examples. It is fast and easy to extend, but it inherits the blind spots of the embedding model, for instance weak handling of negation or of numeric constraints.

An LLM router receives the route descriptions in its prompt and returns a structured choice, usually as JSON or a function call. It handles ambiguous and compound questions better than embeddings, but it adds a full model call to every request and must be constrained so that it cannot invent a route that does not exist.

Query routing is often confused with neighboring techniques because all of them operate on the query before or during retrieval. The distinction lies in what is being changed: the destination, the query text, or the ranking of results.

TechniqueWhat it changesWhen it acts
Query routingDestination of the queryBefore retrieval
Query rewritingWording of the queryBefore retrieval
Hybrid searchCombines dense and sparse scoresDuring retrieval
Metadata filteringSubset of one indexDuring retrieval
RerankingOrder of retrieved candidatesAfter retrieval
How routing differs from neighboring techniques

Metadata filtering and routing can overlap. A filter such as department = legal narrows one shared index, whereas routing selects entirely separate stores or tools. Teams often start with filters and move to routing once sources diverge in format, freshness requirements or access rules.

Designing and tuning a router in practice

A reliable router starts with clear, non-overlapping route definitions. When two routes describe similar content, scores become close and the router behaves unpredictably, so overlapping sources are better merged or separated by a sharper description.

The confidence threshold is the most important tuning parameter. A high threshold sends more queries to the fallback path and reduces wrong routes, while a low threshold answers more queries directly but increases misrouting. Teams usually set it on a labeled evaluation set of several hundred real queries and measure routing accuracy separately from final answer quality.

The decision table below summarizes common design choices.

SituationRecommended approachReason
Few stable sourcesRule-based or semantic routerCheap, fast and predictable
Many overlapping intentsLLM router with structured outputBetter at nuance and compound requests
High traffic, strict latencyTrained classifierLow cost per query at scale
Uncertain or mixed queriesMulti-route dispatch with fusionAvoids losing relevant sources
Sensitive data sourcesRouter plus access checksRouting must not bypass permissions
Choosing a routing strategy

When a query legitimately needs several sources, the router can dispatch it to multiple routes in parallel and merge results with reciprocal rank fusion or a cross-encoder reranker. This multi-route pattern costs more but protects recall on questions that span domains.

Limitations and common failure modes

The central risk of query routing is that errors are silent: a misrouted query still returns fluent text, but it is grounded in the wrong source. Without logging of the chosen route, such failures look like retrieval or generation problems and are hard to diagnose.

  • Route drift: new document types appear, but route descriptions and examples are never updated.
  • Overconfident routing: the router always picks a route even when none fits, so the fallback never triggers.
  • Conversation blindness: follow-up questions such as “and last year?” are routed without the earlier context.
  • Latency stacking: an LLM router placed in front of an already slow pipeline doubles response time.
  • Permission leakage: routing to a privileged tool without checking the user’s access rights.

Good practice is to log every routing decision with its score, review low-confidence cases regularly, and keep a general-purpose fallback route that searches broadly. Routing accuracy should be tracked as its own metric next to RAG evaluation scores.

The short version

Query routing is like the reception desk in a large hospital. You describe your problem once, and the receptionist sends you to the right department instead of making you visit every ward.

In an AI assistant, the router reads the question and decides where the answer most likely lives: a document library, a database, a calculator or a web search. When it chooses well, answers are faster, cheaper and more accurate. When it chooses badly, even a perfect search engine cannot help, because it is searching in the wrong place.

Where it makes a difference

Consider an internal assistant at a mid-sized manufacturing company with three separate knowledge sources: about 8,000 pages of equipment manuals in a vector index, a SQL warehouse with production and sales figures, and an HR policy library. In the first version, every question searched all three sources, and answers to numeric questions often quoted manuals instead of actual figures.

The team added a semantic router with 30 example questions per route and a confidence threshold tuned on 400 logged queries. Questions such as “how many units did line 2 produce in March” now go to a text-to-SQL tool, while maintenance questions go to the manuals index. Low-confidence questions are sent to all three sources and merged with reciprocal rank fusion.

After the change, routing accuracy reached roughly 94 percent on the evaluation set, average response time fell because most queries touched only one source, and the number of answers quoting the wrong source dropped sharply. The logged routing scores also showed which new topics needed a route of their own.