Self-RAG, short for self-reflective retrieval-augmented generation, is a method in which a language model decides on its own when to retrieve external passages and then critiques both the evidence and its own output with special reflection tokens. Within the Retrieval & knowledge area of artificial intelligence, it extends classic RAG by turning retrieval from a fixed preprocessing step into a learned, on-demand decision.
The approach was introduced in 2023 by researchers from the University of Washington and the Allen Institute for AI. Its central idea is that a single model can be trained to generate answers and grade them at the same time, which makes outputs more factual, better grounded and easier to audit than answers produced by a model that always retrieves or never retrieves.
How Self-RAG works during generation
Self-RAG works by interleaving ordinary text tokens with reflection tokens that the model itself emits. Before writing a segment, the model predicts whether retrieval is needed; if it is, a retriever fetches several passages, and the model generates a candidate continuation for each passage in parallel.
Each candidate is then scored by the model’s own critique tokens: is the passage relevant, is the generated sentence supported by it, and how useful is the overall answer. The decoder keeps the best-scoring candidates in a segment-level beam search and moves on to the next segment, so evidence selection happens continuously rather than once at the start.
- The model reads the prompt and the text generated so far.
- It emits a Retrieve token that signals whether new evidence is needed.
- If retrieval is triggered, the retriever returns the top passages, often five to ten.
- The model writes one candidate segment per passage and critiques each one.
- Critique scores rank the candidates, and the best segment is appended to the answer.
The four types of reflection tokens
Reflection tokens are special vocabulary items added to the model so that self-assessment becomes part of normal next-token prediction. The following table lists the four token types defined in the original method and what each one judges.
| Token type | Question it answers | Possible values |
|---|---|---|
| Retrieve | Is external evidence needed now? | Yes, no, continue to use evidence |
| IsRel | Is the retrieved passage relevant? | Relevant, irrelevant |
| IsSup | Does the passage support the generated segment? | Fully, partially, no support |
| IsUse | Is the whole response useful for the query? | Rating from 1 to 5 |
The IsSup token is the most important for factual accuracy, because it separates passages that merely look relevant from passages that actually justify the claim. The IsUse token rewards helpfulness and is computed over the full response, not a single segment.
How Self-RAG is trained
Self-RAG training uses two models and two stages, which keeps the expensive labeling work offline. First, a critic model learns to predict reflection tokens from examples labeled by a stronger proprietary model. Second, the critic annotates a large corpus, and the generator model is trained on that augmented text with a standard language modeling objective.
Because the reflection tokens are inserted into the training data in advance, the generator does not need the critic at inference time. This differs from reinforcement learning from human feedback, where a separate reward model guides optimization; in Self-RAG the feedback is distilled directly into the vocabulary. The published checkpoints were built on Llama 2 at the 7B and 13B parameter scales.
| Stage | Input data | Result |
|---|---|---|
| Critic training | Instruction data with model-labeled reflection tokens | A critic that predicts reflection tokens |
| Corpus augmentation | Instruction-output pairs plus retrieved passages | Text with inline retrieval and critique tokens |
| Generator training | Augmented corpus | One model that retrieves, writes and critiques |
Self-RAG compared with related retrieval methods
Self-RAG belongs to a family of adaptive retrieval methods that try to fix the weaknesses of fixed pipelines. Standard RAG retrieves once for every query, which wastes compute on simple questions and can inject irrelevant context that the model then repeats. The table below contrasts Self-RAG with its closest neighbors.
| Approach | When it retrieves | How quality is controlled |
|---|---|---|
| Standard RAG | Always, once per query | Retriever ranking and optional reranking |
| Self-RAG | On demand, per segment | Learned reflection tokens inside the model |
| Corrective RAG | Always, then evaluates results | External evaluator triggers web search or refinement |
| Agentic RAG | As planned by an agent loop | Tool calls, planning steps and external checks |
| Retrieval-augmented fine-tuning | Always, with distractor documents | Training to ignore irrelevant passages |
The key distinction is where the judgment lives. Corrective RAG and agentic systems rely on external components or prompts, whereas Self-RAG builds the judgment into the weights, so it requires custom training but adds little orchestration code.
Controlling behavior at inference time
A practical advantage of Self-RAG is that its behavior can be tuned without retraining. The probability of the Retrieve token can be compared with an adjustable threshold, and the critique scores can be combined with custom weights during beam search, which lets an operator favor citation precision or fluency for a given task.
| Setting | Typical adjustment | Effect on output |
|---|---|---|
| Retrieval threshold | Lower for fact-heavy tasks | More retrieval, stronger grounding, higher latency |
| IsSup weight | Raise for attribution tasks | Prefers segments fully backed by evidence |
| IsUse weight | Raise for open-ended answers | Prefers fluent, complete responses |
| Beam width | Small values such as 2 or 3 | Wider search improves quality but costs compute |
In the original evaluation, this setup improved results on open-domain question answering, fact verification, reasoning benchmarks and long-form generation with citations, compared with similarly sized models using standard retrieval. Exact gains depend on the retriever, the corpus and the task.
Limitations and common misconceptions
Self-RAG improves grounding but does not guarantee truthfulness. The critique tokens are themselves model predictions, so the system can confidently mislabel an unsupported claim as supported, especially outside the domains seen during training.
- Training cost is real: the method needs a critic, labeled data and full fine-tuning, unlike prompt-only RAG.
- Retriever quality still caps accuracy: if the right passage is never retrieved, reflection cannot recover it.
- Latency grows with parallel candidates, because each retrieved passage produces its own continuation.
- It is not a prompting trick: asking a general model to output similar tags does not reproduce the trained behavior.
- Domain shift weakens the critic, so specialized fields often need new reflection labels.
Stripped to the essentials
Self-RAG is a way of teaching an AI model to check its own homework. Instead of always looking things up or never looking things up, the model first asks itself whether it needs a source. If it does, it reads a few documents, writes a draft based on each one and then grades each draft.
Think of a careful student writing an essay with a stack of library books. The student decides when to open a book, ignores books that are off topic and keeps only the sentences the books really back up. The result is an answer that is more likely to be correct and easier to trace back to its evidence.
Where it is applied
A health information team builds an assistant that answers questions about medication interactions from a library of about 40,000 clinical summaries. A plain RAG prototype retrieves five passages for every question, including greetings and simple follow-ups, and sometimes repeats details from passages that concern a different drug.
The team switches to a Self-RAG model fine-tuned on its own documents. It lowers the retrieval threshold for dosage questions and raises the IsSup weight so that only fully supported sentences survive beam search. Conversational turns no longer trigger retrieval, which cuts the number of search calls noticeably.
In a review of 300 sampled answers, pharmacists find fewer unsupported dosage statements and can trace each remaining claim to a specific passage. The team still keeps human review for high-risk answers, because the critique tokens reduce errors but do not eliminate them.