Agentic RAG lets an AI assistant look things up the way a careful researcher would. Plain RAG (retrieval-augmented generation) searches your documents once and answers from what it finds. With agentic RAG, an AI agent decides what to search for, checks whether the results answer the question, and searches again or elsewhere if they do not. That helps with questions that a single search cannot answer.
How does it differ from plain RAG?
Anthropic puts the difference in one line (How we built our multi-agent research system, June 2025): traditional RAG uses static retrieval, fetching the chunks most similar to the question and answering from those. Its Research feature instead runs a multi-step search that finds relevant information as it goes, adapts to new findings and analyses the results. The survey by Singh et al. (January 2025) defines agentic RAG as RAG with autonomous agents in the pipeline, using reflection, planning, tool use and cooperation between agents to steer retrieval.
Where does the idea come from?
Two research lines led here. In ReAct (Yao et al., 2022), a model alternates reasoning steps with actions, such as lookups through a simple Wikipedia API. On the HotpotQA and Fever question sets, this countered the hallucination and error propagation of chain-of-thought reasoning alone. Self-RAG (Asai et al., 2023) trains a model to retrieve only when needed and to judge both the retrieved passages and its own output, instead of always adding a fixed number of passages.
How does it work?
LangChain's tutorial for a RAG agent shows the basic loop:
- The agent decides whether the question needs a search or can be answered directly.
- It searches with a retriever tool, for example over a vector database.
- A model grades whether the retrieved documents are relevant to the question.
- If they are not, the agent rewrites the question and searches again; if they are, it writes the answer.
The search tool does not have to be a vector search. For structured data, OpenAI advises (March 2026) giving the model a description of the database tables and letting it query the rows it needs, instead of pasting a whole spreadsheet into the prompt. Larger systems split the work: in Anthropic's Research feature a lead agent plans the search and sends out subagents that search in parallel.
Where does it go wrong?
- Cost. In Anthropic's data, agents use about 4× more tokens than chat interactions and multi-agent systems about 15×, so the task has to be worth it. Programmatic tool calling keeps intermediate tool results out of the model's context; in Anthropic's tests (November 2025), average use on complex research tasks fell from 43,588 to 27,297 tokens.
- Overkill. For many applications, Anthropic notes in Building effective agents, a single well-prepared model call with retrieval is enough.
- Open problems. Singh et al. list evaluation, coordination between agents, memory management, efficiency and governance as unresolved.
Example
Suppose an accounting firm asks its assistant: "Which clients paid late more than twice in the past twelve months, and what do our terms say about interest?" Plain RAG searches the documents once. An agentic setup queries the invoicing database for late payments, looks up the interest clause in the standard terms, notices that two clients have their own contract terms, fetches those contracts and only then answers, with a link to each source.
Sources
- Singh et al.: Agentic Retrieval-Augmented Generation, A Survey on Agentic RAG (2025)arXiv
- Anthropic: How we built our multi-agent research systemRSS
- Yao et al.: ReAct, Synergizing Reasoning and Acting in Language Models (2022)arXiv
- Asai et al.: Self-RAG, Learning to Retrieve, Generate, and Critique through Self-Reflection (2023)arXiv
- LangChain: Build a custom RAG agent with LangGraphWeb
- OpenAI: From model to agent, Equipping the Responses API with a computer environmentRSS