Query rewriting is the retrieval step that transforms a user question into one or more better search queries before the retriever runs. In artificial intelligence systems built on retrieval-augmented generation (RAG), it sits between the user and the index and repairs the gap between how people ask and how documents are written.
Within the Retrieval & knowledge toolkit, query rewriting is usually performed by a language model or a rule set that resolves references, removes noise, adds missing context and sometimes splits a question into parts. The goal is higher recall and precision at the retrieval stage, because the generator can only ground its answer in passages that the retriever actually returns.
How query rewriting works in a retrieval pipeline
Query rewriting works by inserting a transformation function between the raw input and the search call. The function receives the user message, often together with conversation history and system metadata, and returns a rewritten query or a set of queries that the retriever then executes against a vector index, a keyword index or both.
In a conversational assistant, the most common job is turning a follow-up into a standalone query. A message such as “what about the second one” carries almost no searchable content, so the rewriter consults earlier turns and produces a self-contained question that names the actual entity. Without this step, dense retrieval embeds a vague fragment and returns passages that match the wording rather than the intent.
The typical stages are shown below, from the user message to the retrieved candidate set.
| Stage | What happens | Typical output |
|---|---|---|
| Context resolution | Pronouns and ellipses are replaced with named entities | A standalone question |
| Normalization | Typos, filler and chat formatting are removed | A clean, compact query |
| Enrichment | Synonyms, product names or date ranges are added | A query with domain vocabulary |
| Decomposition | A compound question is split into sub-questions | Two to five focused queries |
| Formatting | The query is adapted to the target index or filter syntax | Keyword string, vector text or filter |
Not every pipeline uses every stage. A simple document search may only need normalization, while a multi-hop retrieval system depends heavily on decomposition so that each hop has a precise target.
Main techniques and variants
Query rewriting covers a family of techniques that differ in what the model produces and how many retrieval calls follow. The choice affects latency, cost and the kind of failure that remains.
| Technique | Output | Best suited for |
|---|---|---|
| Standalone rewriting | One self-contained question | Multi-turn chat assistants |
| Multi-query generation | Several paraphrases of one question | Ambiguous or broad questions |
| Sub-question decomposition | Independent sub-questions | Comparisons and multi-hop questions |
| Step-back prompting | A more general background question | Questions that need principles first |
| Hypothetical document generation | A draft answer used as the search text | Short queries against long documents |
| Structured extraction | Query text plus metadata filters | Catalogs, logs and dated records |
Multi-query generation produces several phrasings and merges their result lists, often with reciprocal rank fusion, which raises recall when the vocabulary of the corpus is unpredictable. Hypothetical document generation, known as HyDE, embeds a model-written pseudo-answer instead of the question, because a draft answer tends to sit closer to real answers in embedding space. Structured extraction turns phrases such as “invoices from last quarter” into an explicit date filter, which pure semantic search handles poorly.
Query rewriting compared with related techniques
Query rewriting is often confused with neighboring steps in the retrieval stack, but it operates on the query before search, while several related techniques operate on documents or on results after search.
| Technique | Acts on | Main purpose |
|---|---|---|
| Query rewriting | The query, before retrieval | Reformulate intent into searchable form |
| Query expansion | The query, before retrieval | Add related terms to the original wording |
| Query routing | The query, before retrieval | Choose which index or tool to search |
| Reranking | Candidate results, after retrieval | Reorder results by fine-grained relevance |
| Contextual retrieval | Document chunks, at indexing time | Add context to chunks before embedding |
Query expansion is the narrower, older idea from classic information retrieval: it keeps the original query and appends terms, for example through pseudo-relevance feedback with BM25. Query rewriting may replace the wording entirely. In practice, mature systems combine the steps, rewriting first, routing second, retrieving with hybrid search and finally reranking with a cross-encoder.
Designing and tuning a rewriting step
A good rewriting step is designed against measured retrieval quality, not against how natural the rewritten text looks. Teams usually build a set of real user questions with known relevant passages and compare recall at k and mean reciprocal rank with and without the rewriter.
The following decision table summarizes common situations and the rewriting strategy that typically fits them.
| Situation | Recommended strategy | Reason |
|---|---|---|
| Follow-up questions in chat | Standalone rewriting with recent turns | Restores missing entities |
| Short keyword queries | Multi-query generation or HyDE | Adds vocabulary the corpus uses |
| Questions comparing two items | Sub-question decomposition | Retrieves evidence for each item |
| Dates, versions or categories in text | Structured extraction into filters | Applies exact constraints |
| Strict latency budget | Small model or rule-based rewriting | Keeps added delay low |
Latency matters because rewriting adds a full model call before search. A small instruction-tuned model often adds roughly 100 to 500 milliseconds, depending on hardware and output length, while a large hosted model can add more than a second. Many teams therefore skip rewriting for first-turn queries that are already specific and apply it only to follow-ups or short inputs.
The rewriting prompt should instruct the model to preserve exact identifiers, such as error codes, part numbers and quoted phrases, and to avoid answering the question. Logging both the original and the rewritten query is essential, because retrieval failures are invisible without that trace.
Limitations, risks and common mistakes
The central risk of query rewriting is intent drift: the rewritten query can be fluent yet ask something different from what the user meant. Because the generator never sees the original retrieval target, the final answer may be confidently grounded in the wrong passages.
| Symptom | Likely cause | Fix |
|---|---|---|
| Exact codes no longer match | Rewriter paraphrased identifiers | Instruct the model to copy identifiers verbatim |
| Results ignore earlier context | History window too short | Pass the relevant recent turns |
| Answers drift to general topics | Over-broad step-back query | Search with original and rewritten queries |
| Facts appear in the query | Rewriter answered instead of rephrasing | Forbid answers in the prompt and validate output |
| Slow responses | Large model used for every query | Use a smaller model or conditional rewriting |
- Hallucinated constraints, such as invented dates or product names, can silently filter out the correct documents.
- Searching only with the rewritten query removes a safety net; keeping the original query in the candidate set is a cheap safeguard.
- Evaluation that reads rewritten queries by eye rather than measuring recall tends to reward fluent but unhelpful rewrites.
In simple terms
Query rewriting is like a helpful librarian who listens to a vague question and turns it into a precise request before going to the shelves. If someone asks “and the newer one?”, the librarian remembers the earlier conversation and searches for the exact book that was meant.
For an AI assistant, this means the search step gets a clear question instead of a messy one. Better questions bring back better documents, and better documents lead to better answers.
A practical example
A software company runs an internal support assistant over about 40,000 pages of product documentation. Engineers often type follow-ups such as “does that also work on version 7?” after asking about a configuration flag. Without rewriting, the retriever embeds that fragment and returns generic upgrade notes.
The team adds a small model that rewrites each follow-up into a standalone question naming the flag and the version, extracts “version 7” as a metadata filter and keeps flag names verbatim. On a test set of 300 real questions with labeled answers, recall at 10 rises from about 0.62 to 0.81, and the added delay stays near 200 milliseconds. Support staff report far fewer answers about the wrong release.