Retrieval-augmented generation (RAG) is a technique in which a system retrieves relevant passages from an external knowledge source and inserts them into the prompt of a language model, which then generates an answer grounded in that material. RAG separates what the model knows from what the system can look up, so answers can reflect private, specialized or recently updated information without retraining the model.
In artificial intelligence, RAG has become the default architecture for question answering over company documents, support knowledge bases, legal and technical archives and product catalogs. Its quality depends far more on the retrieval step than on the size of the generating model: a model cannot ground its answer in a passage that was never retrieved.
How a RAG pipeline works
A RAG system runs in two phases: an offline indexing phase that prepares the knowledge source, and an online query phase that runs for every user question. The indexing phase is done once and refreshed when documents change; the query phase must be fast, usually well under a few seconds end to end.
The table walks through both phases in order.
| Phase | Stage | What happens |
|---|---|---|
| Indexing | Ingestion and parsing | Documents are converted to clean text with structure and metadata |
| Indexing | Chunking | Text is split into passages small enough to retrieve precisely |
| Indexing | Embedding and storage | Each chunk becomes a vector stored in a vector index with its metadata |
| Query | Retrieval | The question is embedded and the most similar chunks are found |
| Query | Reranking | A stronger model reorders candidates and keeps the best few |
| Query | Augmented generation | Selected chunks and the question go into the prompt; the model answers with references |
The prompt typically instructs the model to answer only from the supplied context and to say when the context does not contain the answer. That instruction, combined with references to the retrieved chunks, lets users verify each claim.
Retrieval methods used in RAG
The retriever decides which evidence the model ever sees, so retrieval quality sets the ceiling for answer quality. Production systems rarely rely on a single method.
| Method | How it matches | Best at | Weak at |
|---|---|---|---|
| Dense vector search | Similarity of embeddings, usually cosine | Paraphrases and conceptual questions | Exact codes, rare names, numbers |
| Sparse keyword search (BM25) | Weighted term overlap | Product codes, error messages, names | Synonyms and reworded questions |
| Hybrid search | Combines dense and sparse results, often by rank fusion | Mixed real-world queries | Needs tuning of two indexes |
| Cross-encoder reranking | Scores each question and chunk pair jointly | Precision in the final top results | Too slow for the full corpus |
| Metadata filtering | Restricts search by date, product, language or access rights | Scoped and permission-aware answers | Depends on clean metadata |
A common and robust pattern is hybrid retrieval of 20 to 50 candidates followed by reranking down to 3 to 10 chunks. Reranking is affordable here because it scores only the shortlist, not the whole index.
Key design parameters
Most RAG quality problems trace back to a few tunable parameters. Their best values depend on the documents and questions, so they should be set by measurement on a test set rather than copied from defaults.
| Parameter | Typical range | Effect of increasing it |
|---|---|---|
| Chunk size | 200 to 1,000 tokens | More context per hit, less precise matching |
| Chunk overlap | About 10 to 20 percent of chunk size | Fewer facts cut at boundaries, larger index |
| Top-k retrieved | 3 to 50 candidates | Higher recall, more noise and token cost |
| Embedding dimension | About 384 to 3,072 | Richer representation, more storage and slower search |
| Similarity threshold | Model-specific cutoff score | Fewer irrelevant chunks, more unanswered questions |
Structure-aware chunking, which splits at headings, paragraphs or table boundaries instead of at a fixed token count, usually beats naive splitting. Storing the document title and section path with each chunk also helps both retrieval and the citations shown to users.
RAG compared with fine-tuning and long context
RAG, fine-tuning and very long context windows are often presented as alternatives, but they solve different problems. RAG supplies knowledge, fine-tuning shapes behavior and style, and a long context window simply raises how much text one request can hold.
| Approach | Best for | Update cost | Main limitation |
|---|---|---|---|
| RAG | Large, changing or access-controlled knowledge | Re-index changed documents only | Fails when retrieval misses the evidence |
| Fine-tuning | Consistent format, tone or domain skill | New training run per update | Poor at memorizing exact, changing facts |
| Long context | Analysis of a few known documents at once | None, documents sent per request | Cost and latency grow with every token |
The approaches combine well. A fine-tuned model can be better at using retrieved passages faithfully, and a long context window lets RAG pass more candidates when a question needs broad evidence.
Failure modes and evaluation
A RAG answer can go wrong at retrieval or at generation, and the two must be measured separately to know what to fix. Typical failures include:
- Retrieval miss: the relevant passage exists but is not in the top results, often because of vocabulary mismatch.
- Broken context: chunking separated a fact from the heading or table that gives it meaning.
- Stale or conflicting sources: the index holds outdated versions next to current ones.
- Unfaithful generation: the model ignores the context or adds unsupported details.
- Indirect prompt injection: a retrieved document contains instructions that the model follows.
Good teams build a labeled evaluation set of real questions and track retrieval recall at k, context precision, answer faithfulness to the retrieved text and answer relevance. Access rights must also be enforced at retrieval time, because a model cannot be trusted to hide a passage it has already been given.
Put simply
RAG is like letting a knowledgeable assistant look things up before answering. Instead of replying from memory alone, the system first searches a trusted collection of documents, picks the most relevant pages and hands them to the AI together with the question.
The AI then writes its answer from those pages and can point to where each fact came from. The answer is only as good as the pages it finds, so the search step matters as much as the AI itself.
In practice
A manufacturing company with about 8,000 pages of maintenance manuals wants technicians to get answers on a tablet at the machine. A first RAG prototype uses fixed 1,000-token chunks and dense vector search only, and on a test set of 300 real technician questions it finds the right passage in the top five results for about 70 percent of them.
The team makes three changes. It chunks the manuals by section at about 500 tokens with a 50-token overlap, adds keyword search for part numbers and error codes, and reranks the top 30 hybrid results down to five. Each chunk now carries the machine model as metadata, so a question about one press never retrieves the manual for another.
Retrieval of the correct passage in the top five rises to about 90 percent, and every answer shows the manual page it came from. Technicians trust the tool because they can check the source in one tap, and the company updates answers simply by re-indexing revised manuals.