Agentic RAG

On this page 7

Agentic RAG is a form of retrieval-augmented generation in which a language model acts as an autonomous agent that decides when, where and how to retrieve information, evaluates what it finds and repeats the process until it has enough evidence to answer. Within the Retrieval & knowledge domain of artificial intelligence, it extends classic RAG from a fixed retrieve-then-generate pipeline into a reasoning loop with tools, planning and self-verification.

The approach targets questions that a single search cannot answer: multi-hop questions, comparisons across sources, tasks that combine documents with databases or APIs, and queries whose wording does not match the vocabulary of the indexed content. The price is higher latency, higher token cost and more complex failure modes than a standard pipeline.

How agentic RAG works as a control loop

Agentic RAG replaces the single retrieval call with a plan, act, observe and reflect cycle driven by the model itself. The agent reads the user request, decomposes it into sub-questions, chooses a retrieval tool for each, inspects the returned passages and decides whether to search again, switch sources or stop.

The loop is usually implemented through tool calling: retrieval functions such as vector search, keyword search, SQL queries or web search are exposed to the model with a schema, and the model emits structured calls. Patterns such as ReAct-style reasoning, where thoughts and actions alternate, are common foundations. An orchestrator executes the calls, enforces limits and returns observations to the model’s context.

The following table lists the typical stages of one agentic retrieval cycle and what each stage contributes.

StageWhat the agent doesTypical technique
RoutingDecides whether retrieval is needed and which source fitsClassifier prompt or tool selection
DecompositionSplits a complex question into sub-queriesPlanning prompt, query rewriting
RetrievalRuns searches against one or more indexesHybrid search, BM25, vector similarity
EvaluationGrades relevance and sufficiency of resultsLLM-as-judge, reranking scores
IterationReformulates queries or tries another sourceSelf-correction, fallback to web search
SynthesisWrites the answer with citations to passagesGrounded generation with attribution
Stages of an agentic retrieval cycle

Architectural variants of agentic RAG

Agentic RAG is a family of designs rather than one algorithm, and the variants differ mainly in how much autonomy the model receives. A single-agent router may only choose among indexes, while a multi-agent system can assign separate agents to planning, retrieval, verification and writing.

The table compares the most frequently deployed variants by autonomy and typical use.

VariantDegree of autonomyBest suited for
Routing RAGLow: one decision per querySeveral separate knowledge bases
Corrective RAGMedium: grades and retries retrievalNoisy or incomplete indexes
Self-reflective RAGMedium: critiques its own draftHigh-accuracy question answering
Iterative multi-hop RAGHigh: chains dependent searchesQuestions needing linked facts
Multi-agent RAGVery high: specialized cooperating agentsResearch reports and complex analysis
Common agentic RAG variants

Variants are often combined. A research assistant might use routing for source selection, corrective grading for each result set and a reflection step before the final answer. Some systems also query a knowledge graph for entity relations, which overlaps with GraphRAG, while keeping unstructured passages in a vector index.

Agentic RAG compared with classic RAG

The essential difference is that classic RAG retrieves once with the original query, whereas agentic RAG treats retrieval as a decision the model can revisit. Classic pipelines are predictable and cheap; agentic ones are adaptive but harder to bound.

The comparison below summarizes the practical consequences of that difference for engineering teams.

PropertyClassic RAGAgentic RAG
Retrieval calls per queryUsually oneOften 2 to 10 or more
Query formulationUser wording, sometimes rewritten onceRepeatedly rewritten by the agent
SourcesOne indexMultiple indexes, tools and APIs
LatencyRoughly 1 to 3 secondsSeveral seconds to minutes
Cost per answerLow and predictableHigher and variable
Handling of missing evidenceAnswers anyway or refusesSearches further or reports the gap
DebuggingLinear traceBranching trace with many steps
Classic and agentic RAG side by side

Neither design is universally superior. For frequently asked, single-fact questions, a well-tuned classic pipeline with good chunking and reranking often matches agentic quality at a fraction of the cost. Agentic RAG earns its overhead when questions are compositional or when the right source is not known in advance.

Designing and configuring an agentic RAG system

A robust implementation starts with strong retrieval primitives, because an agent cannot reason its way past a poor index. Clean chunking, sound embeddings, hybrid search and a reranker remain the foundation; the agentic layer only decides how to use them.

  • Define narrow, well-described tools, each with a clear purpose, input schema and example queries.
  • Set hard budgets: a maximum number of steps, tokens and wall-clock time per request.
  • Add an explicit sufficiency check so the agent stops when evidence is adequate instead of searching indefinitely.
  • Require passage-level citations in the final answer to support grounding and review.
  • Log every tool call and observation to enable tracing, evaluation and cost analysis.

The decision table below helps match the design to the workload.

SituationRecommended designReason
Single index, factual lookupsClassic RAG with rerankingLowest cost and latency
Several domain indexesRouting RAGOne cheap decision improves precision
Questions needing chained factsIterative multi-hop RAGLater queries depend on earlier answers
Low-quality or sparse corpusCorrective RAG with fallbackDetects weak results and retries
Long analytical reportsMulti-agent RAGDivides planning, research and writing
Choosing a retrieval design

Limitations, risks and evaluation of agentic RAG

The main risks of agentic RAG come from compounding errors across steps. A wrong sub-question early in the plan steers every later search, and an agent may loop on reformulations without progress or stop too early with confident but partial evidence.

Security is a specific concern. Retrieved documents can contain indirect prompt injection, text that tries to redirect the agent toward unintended tool calls. Mitigations include treating retrieved content strictly as data, restricting tool permissions, and requiring human-in-the-loop approval for actions with side effects.

Evaluation must therefore cover the trajectory, not only the final answer. Teams typically measure answer correctness and faithfulness, retrieval recall across all steps, average number of tool calls and cost per resolved query, and compare the agentic system against a classic baseline on the same question set.

In everyday language

A normal AI search assistant looks something up once and then answers with whatever it found. Agentic RAG works more like a careful researcher. It decides what it needs to know, looks in the right places, reads the results, and searches again if something is missing.

Think of the difference between glancing at one encyclopedia page and visiting a library where you check several books, ask a librarian and compare notes. The answer is usually better, but it takes longer and costs more. That trade-off is the heart of the idea.

Seen in the real world

A legal operations team at a mid-sized manufacturer receives questions such as which supplier contracts signed after 2022 allow termination without penalty and also lack a data protection clause. A classic single-pass pipeline retrieves a handful of passages that mention termination and misses most of the relevant agreements.

The team deploys an agentic RAG assistant with three tools: a SQL query over the contract registry, a hybrid search over clause text and a clause classifier. The agent first filters contracts by date in the registry, then searches each candidate for termination and data protection clauses, and finally checks borderline wording a second time before answering.

A typical query now takes about 40 seconds and eight tool calls instead of three seconds, but in the team’s internal review the list of matching contracts was complete, with every entry linked to the cited clause, replacing roughly a day of manual reading.