Query rewriting · retrieval

On this page 7

Query rewriting is the retrieval step that transforms a user question into one or more better search queries before the retriever runs. In artificial intelligence systems built on retrieval-augmented generation (RAG), it sits between the user and the index and repairs the gap between how people ask and how documents are written.

Within the Retrieval & knowledge toolkit, query rewriting is usually performed by a language model or a rule set that resolves references, removes noise, adds missing context and sometimes splits a question into parts. The goal is higher recall and precision at the retrieval stage, because the generator can only ground its answer in passages that the retriever actually returns.

How query rewriting works in a retrieval pipeline

Query rewriting works by inserting a transformation function between the raw input and the search call. The function receives the user message, often together with conversation history and system metadata, and returns a rewritten query or a set of queries that the retriever then executes against a vector index, a keyword index or both.

In a conversational assistant, the most common job is turning a follow-up into a standalone query. A message such as “what about the second one” carries almost no searchable content, so the rewriter consults earlier turns and produces a self-contained question that names the actual entity. Without this step, dense retrieval embeds a vague fragment and returns passages that match the wording rather than the intent.

The typical stages are shown below, from the user message to the retrieved candidate set.

StageWhat happensTypical output
Context resolutionPronouns and ellipses are replaced with named entitiesA standalone question
NormalizationTypos, filler and chat formatting are removedA clean, compact query
EnrichmentSynonyms, product names or date ranges are addedA query with domain vocabulary
DecompositionA compound question is split into sub-questionsTwo to five focused queries
FormattingThe query is adapted to the target index or filter syntaxKeyword string, vector text or filter
Stages of a query rewriting step

Not every pipeline uses every stage. A simple document search may only need normalization, while a multi-hop retrieval system depends heavily on decomposition so that each hop has a precise target.

Main techniques and variants

Query rewriting covers a family of techniques that differ in what the model produces and how many retrieval calls follow. The choice affects latency, cost and the kind of failure that remains.

TechniqueOutputBest suited for
Standalone rewritingOne self-contained questionMulti-turn chat assistants
Multi-query generationSeveral paraphrases of one questionAmbiguous or broad questions
Sub-question decompositionIndependent sub-questionsComparisons and multi-hop questions
Step-back promptingA more general background questionQuestions that need principles first
Hypothetical document generationA draft answer used as the search textShort queries against long documents
Structured extractionQuery text plus metadata filtersCatalogs, logs and dated records
Common query rewriting techniques

Multi-query generation produces several phrasings and merges their result lists, often with reciprocal rank fusion, which raises recall when the vocabulary of the corpus is unpredictable. Hypothetical document generation, known as HyDE, embeds a model-written pseudo-answer instead of the question, because a draft answer tends to sit closer to real answers in embedding space. Structured extraction turns phrases such as “invoices from last quarter” into an explicit date filter, which pure semantic search handles poorly.

Query rewriting is often confused with neighboring steps in the retrieval stack, but it operates on the query before search, while several related techniques operate on documents or on results after search.

TechniqueActs onMain purpose
Query rewritingThe query, before retrievalReformulate intent into searchable form
Query expansionThe query, before retrievalAdd related terms to the original wording
Query routingThe query, before retrievalChoose which index or tool to search
RerankingCandidate results, after retrievalReorder results by fine-grained relevance
Contextual retrievalDocument chunks, at indexing timeAdd context to chunks before embedding
Query rewriting and neighboring techniques

Query expansion is the narrower, older idea from classic information retrieval: it keeps the original query and appends terms, for example through pseudo-relevance feedback with BM25. Query rewriting may replace the wording entirely. In practice, mature systems combine the steps, rewriting first, routing second, retrieving with hybrid search and finally reranking with a cross-encoder.

Designing and tuning a rewriting step

A good rewriting step is designed against measured retrieval quality, not against how natural the rewritten text looks. Teams usually build a set of real user questions with known relevant passages and compare recall at k and mean reciprocal rank with and without the rewriter.

The following decision table summarizes common situations and the rewriting strategy that typically fits them.

SituationRecommended strategyReason
Follow-up questions in chatStandalone rewriting with recent turnsRestores missing entities
Short keyword queriesMulti-query generation or HyDEAdds vocabulary the corpus uses
Questions comparing two itemsSub-question decompositionRetrieves evidence for each item
Dates, versions or categories in textStructured extraction into filtersApplies exact constraints
Strict latency budgetSmall model or rule-based rewritingKeeps added delay low
Choosing a rewriting strategy

Latency matters because rewriting adds a full model call before search. A small instruction-tuned model often adds roughly 100 to 500 milliseconds, depending on hardware and output length, while a large hosted model can add more than a second. Many teams therefore skip rewriting for first-turn queries that are already specific and apply it only to follow-ups or short inputs.

The rewriting prompt should instruct the model to preserve exact identifiers, such as error codes, part numbers and quoted phrases, and to avoid answering the question. Logging both the original and the rewritten query is essential, because retrieval failures are invisible without that trace.

Limitations, risks and common mistakes

The central risk of query rewriting is intent drift: the rewritten query can be fluent yet ask something different from what the user meant. Because the generator never sees the original retrieval target, the final answer may be confidently grounded in the wrong passages.

SymptomLikely causeFix
Exact codes no longer matchRewriter paraphrased identifiersInstruct the model to copy identifiers verbatim
Results ignore earlier contextHistory window too shortPass the relevant recent turns
Answers drift to general topicsOver-broad step-back querySearch with original and rewritten queries
Facts appear in the queryRewriter answered instead of rephrasingForbid answers in the prompt and validate output
Slow responsesLarge model used for every queryUse a smaller model or conditional rewriting
Typical failures and their fixes
  • Hallucinated constraints, such as invented dates or product names, can silently filter out the correct documents.
  • Searching only with the rewritten query removes a safety net; keeping the original query in the candidate set is a cheap safeguard.
  • Evaluation that reads rewritten queries by eye rather than measuring recall tends to reward fluent but unhelpful rewrites.

In simple terms

Query rewriting is like a helpful librarian who listens to a vague question and turns it into a precise request before going to the shelves. If someone asks “and the newer one?”, the librarian remembers the earlier conversation and searches for the exact book that was meant.

For an AI assistant, this means the search step gets a clear question instead of a messy one. Better questions bring back better documents, and better documents lead to better answers.

A practical example

A software company runs an internal support assistant over about 40,000 pages of product documentation. Engineers often type follow-ups such as “does that also work on version 7?” after asking about a configuration flag. Without rewriting, the retriever embeds that fragment and returns generic upgrade notes.

The team adds a small model that rewrites each follow-up into a standalone question naming the flag and the version, extracts “version 7” as a metadata filter and keeps flag names verbatim. On a test set of 300 real questions with labeled answers, recall at 10 rises from about 0.62 to 0.81, and the added delay stays near 200 milliseconds. Support staff report far fewer answers about the wrong release.