Self-RAG

On this page 8

Self-RAG, short for self-reflective retrieval-augmented generation, is a method in which a language model decides on its own when to retrieve external passages and then critiques both the evidence and its own output with special reflection tokens. Within the Retrieval & knowledge area of artificial intelligence, it extends classic RAG by turning retrieval from a fixed preprocessing step into a learned, on-demand decision.

The approach was introduced in 2023 by researchers from the University of Washington and the Allen Institute for AI. Its central idea is that a single model can be trained to generate answers and grade them at the same time, which makes outputs more factual, better grounded and easier to audit than answers produced by a model that always retrieves or never retrieves.

How Self-RAG works during generation

Self-RAG works by interleaving ordinary text tokens with reflection tokens that the model itself emits. Before writing a segment, the model predicts whether retrieval is needed; if it is, a retriever fetches several passages, and the model generates a candidate continuation for each passage in parallel.

Each candidate is then scored by the model’s own critique tokens: is the passage relevant, is the generated sentence supported by it, and how useful is the overall answer. The decoder keeps the best-scoring candidates in a segment-level beam search and moves on to the next segment, so evidence selection happens continuously rather than once at the start.

  1. The model reads the prompt and the text generated so far.
  2. It emits a Retrieve token that signals whether new evidence is needed.
  3. If retrieval is triggered, the retriever returns the top passages, often five to ten.
  4. The model writes one candidate segment per passage and critiques each one.
  5. Critique scores rank the candidates, and the best segment is appended to the answer.

The four types of reflection tokens

Reflection tokens are special vocabulary items added to the model so that self-assessment becomes part of normal next-token prediction. The following table lists the four token types defined in the original method and what each one judges.

Token typeQuestion it answersPossible values
RetrieveIs external evidence needed now?Yes, no, continue to use evidence
IsRelIs the retrieved passage relevant?Relevant, irrelevant
IsSupDoes the passage support the generated segment?Fully, partially, no support
IsUseIs the whole response useful for the query?Rating from 1 to 5
Reflection tokens in Self-RAG

The IsSup token is the most important for factual accuracy, because it separates passages that merely look relevant from passages that actually justify the claim. The IsUse token rewards helpfulness and is computed over the full response, not a single segment.

How Self-RAG is trained

Self-RAG training uses two models and two stages, which keeps the expensive labeling work offline. First, a critic model learns to predict reflection tokens from examples labeled by a stronger proprietary model. Second, the critic annotates a large corpus, and the generator model is trained on that augmented text with a standard language modeling objective.

Because the reflection tokens are inserted into the training data in advance, the generator does not need the critic at inference time. This differs from reinforcement learning from human feedback, where a separate reward model guides optimization; in Self-RAG the feedback is distilled directly into the vocabulary. The published checkpoints were built on Llama 2 at the 7B and 13B parameter scales.

StageInput dataResult
Critic trainingInstruction data with model-labeled reflection tokensA critic that predicts reflection tokens
Corpus augmentationInstruction-output pairs plus retrieved passagesText with inline retrieval and critique tokens
Generator trainingAugmented corpusOne model that retrieves, writes and critiques
Training stages and their outputs

Self-RAG belongs to a family of adaptive retrieval methods that try to fix the weaknesses of fixed pipelines. Standard RAG retrieves once for every query, which wastes compute on simple questions and can inject irrelevant context that the model then repeats. The table below contrasts Self-RAG with its closest neighbors.

ApproachWhen it retrievesHow quality is controlled
Standard RAGAlways, once per queryRetriever ranking and optional reranking
Self-RAGOn demand, per segmentLearned reflection tokens inside the model
Corrective RAGAlways, then evaluates resultsExternal evaluator triggers web search or refinement
Agentic RAGAs planned by an agent loopTool calls, planning steps and external checks
Retrieval-augmented fine-tuningAlways, with distractor documentsTraining to ignore irrelevant passages
Self-RAG and neighboring approaches

The key distinction is where the judgment lives. Corrective RAG and agentic systems rely on external components or prompts, whereas Self-RAG builds the judgment into the weights, so it requires custom training but adds little orchestration code.

Controlling behavior at inference time

A practical advantage of Self-RAG is that its behavior can be tuned without retraining. The probability of the Retrieve token can be compared with an adjustable threshold, and the critique scores can be combined with custom weights during beam search, which lets an operator favor citation precision or fluency for a given task.

SettingTypical adjustmentEffect on output
Retrieval thresholdLower for fact-heavy tasksMore retrieval, stronger grounding, higher latency
IsSup weightRaise for attribution tasksPrefers segments fully backed by evidence
IsUse weightRaise for open-ended answersPrefers fluent, complete responses
Beam widthSmall values such as 2 or 3Wider search improves quality but costs compute
Inference settings and their effect

In the original evaluation, this setup improved results on open-domain question answering, fact verification, reasoning benchmarks and long-form generation with citations, compared with similarly sized models using standard retrieval. Exact gains depend on the retriever, the corpus and the task.

Limitations and common misconceptions

Self-RAG improves grounding but does not guarantee truthfulness. The critique tokens are themselves model predictions, so the system can confidently mislabel an unsupported claim as supported, especially outside the domains seen during training.

  • Training cost is real: the method needs a critic, labeled data and full fine-tuning, unlike prompt-only RAG.
  • Retriever quality still caps accuracy: if the right passage is never retrieved, reflection cannot recover it.
  • Latency grows with parallel candidates, because each retrieved passage produces its own continuation.
  • It is not a prompting trick: asking a general model to output similar tags does not reproduce the trained behavior.
  • Domain shift weakens the critic, so specialized fields often need new reflection labels.

Stripped to the essentials

Self-RAG is a way of teaching an AI model to check its own homework. Instead of always looking things up or never looking things up, the model first asks itself whether it needs a source. If it does, it reads a few documents, writes a draft based on each one and then grades each draft.

Think of a careful student writing an essay with a stack of library books. The student decides when to open a book, ignores books that are off topic and keeps only the sentences the books really back up. The result is an answer that is more likely to be correct and easier to trace back to its evidence.

Where it is applied

A health information team builds an assistant that answers questions about medication interactions from a library of about 40,000 clinical summaries. A plain RAG prototype retrieves five passages for every question, including greetings and simple follow-ups, and sometimes repeats details from passages that concern a different drug.

The team switches to a Self-RAG model fine-tuned on its own documents. It lowers the retrieval threshold for dosage questions and raises the IsSup weight so that only fully supported sentences survive beam search. Conversational turns no longer trigger retrieval, which cuts the number of search calls noticeably.

In a review of 300 sampled answers, pharmacists find fewer unsupported dosage statements and can trace each remaining claim to a specific passage. The team still keeps human review for high-risk answers, because the critique tokens reduce errors but do not eliminate them.