Agentic RAG: What It Is and When to Use It

From a fixed pipeline to a decision loop

A plain RAG pipeline does the same thing every time. It takes the question, searches an index, builds a prompt and generates an answer. The path is decided in advance by the developer, and it does not change with the question. If you are new to the pattern, start with what RAG is and when to use it.

Agentic RAG hands some of those decisions to a model at run time. The retriever becomes a tool that the model can choose to call, usually within a loop. The model looks at the question, and perhaps at earlier results, then chooses to search, search differently, use another source or answer. LangGraph's documentation has a tutorial along these lines, in which the model either replies directly or calls a retriever tool, and a weak result leads to a rewritten question and another attempt.

"Agentic RAG" is a descriptive label and not a formal standard, so people use it for systems of differing complexity. A helpful test is to ask who chooses the next step. If your code always follows the same path, it is a pipeline, however many stages it has. If a model's output can change the path, it has an agentic element.

What the model can decide

Agency is a matter of degree, and you can grant it one decision at a time. These are the common ones, roughly from least to most autonomy:

Whether to retrieve. A greeting or a question about the conversation itself may need no search.
Where to retrieve from. A router sends the question to the product documentation, a contracts index or a database.
How to query. The model rewrites a vague question or fills in details from the conversation.
How to divide the question. A comparison between two plans may become one search for each.
Whether the results are sufficient. The model or a separate grader judges the passages, and the system tries again if they fall short.
Whether the answer is supported. A final check compares the draft with its sources before it is returned.

Research has explored versions of these ideas. The Corrective RAG paper uses an evaluator to grade retrieved passages and trigger follow-up actions, and the Self-RAG paper trains a model to decide when to retrieve and to critique passages and its own output. Neither one defines agentic RAG, but both show the appeal of treating retrieval as something to assess rather than simply trust.

What the loop looks like

The following sample is illustrative. All the helper functions are placeholders for your own code, and it runs no real model or search. What matters is the shape, especially the limit on attempts and the honest way out when that limit is reached:

# Illustrative only. All helper functions are placeholders.
MAX_SEARCHES = 3

def answer(question):
    gathered = []
    query = question

    for _ in range(MAX_SEARCHES):
        source = choose_source(query)
        gathered += search(source, query)

        verdict = assess(question, gathered)
        if verdict.sufficient:
            return generate(question, gathered)
        query = rewrite(question, verdict.missing)

    return "I could not find enough information to answer that."

This version keeps the loop in ordinary code, with the model filling in specific decisions. Alternatively, you can give the model a search tool and let it issue calls until it replies, which is more flexible but harder to predict. In either version, the limit belongs in your code and not in the prompt. The guide to tool calling for AI agents covers tool descriptions, validation and loop limits in more depth.

When it beats a plain pipeline

The extra machinery earns its place when a single search cannot reliably gather what the question needs. Good signs include:

Multi-part questions. "How do the refund terms differ between these two plans?" may need separate searches, because a single query can match one plan and miss the other.
Dependent steps. When the second search depends on what the first one found, the query cannot be written in advance. An example is finding which team owns a service and then looking up that team's process.
Several kinds of source. One question may need a document index, while the next needs a database or a live API. A router, or a set of tools, handles this variety.
Uneven retrieval. If failures come largely from poor queries, an assess-and-retry step can recover some of them. Your logs and evaluation should show that this is the cause.
Tolerance for delay. Research-style tasks, where a user expects to wait, are a better fit than an instant help widget.

When a plain pipeline is the better choice

Many questions are direct lookups with an answer in one place. For these, an agent adds cost and new ways to fail without improving the result. Stay with a fixed pipeline if:

most test questions can be answered from a single good search;
failures come from parsing, chunking or search quality, which a loop will not cure, and may simply repeat;
you need consistent latency and cost per request;
you have to explain every step of the process, and a predictable path makes this easier.

It also helps to separate retrieval problems from control problems. Hybrid search and reranking can fix vocabulary and ranking issues in a fixed pipeline. Query rewriting can be a fixed step too, without any decision loop. Try improvements of this kind first, since they keep the system predictable.

What agency costs

Every decision the model makes is another call and another chance to go wrong. Plan for the following:

Cost and latency. Routing, grading and rewriting each add model calls, and the gathered context grows with every search. See controlling LLM cost and latency for ways to limit this.
Unpredictable paths. The same question may take different routes on different runs, which makes failures harder to reproduce. Log every decision, query and result for each run.
Loops that go nowhere. A model may repeat near-identical searches. Limit the attempts, and consider stopping when a new search returns nothing new.
Graders that are wrong. A relevance check can reject good passages or approve bad ones, and it needs testing like any other component.
A larger attack surface. Retrieved text can contain instructions, and it now reaches a model that controls further actions. Restrict tools to read-only access where possible, and enforce permissions outside the model. See LLM guardrails and prompt injection defence.

Keeping the agent's permissions small

Because agentic retrieval usually needs only reading, it is one of the safer places to give a model some autonomy. Keep it that way. If the same agent can also send messages or change records, a poisoned document becomes much more dangerous, so separate the retrieving agent from anything that acts.

How to test it

Evaluate the system as a set of decisions as well as an end result. Alongside the usual retrieval and answer measures described in how to evaluate a RAG system, track these for each question:

whether routing selected a suitable source;
how many searches and model calls were made, and the total latency and cost;
how often retries improved the result, and how often they only added cost;
how often the system correctly declined when no answer was available.

Most usefully, run your fixed pipeline on the same test set as a baseline. Agentic RAG should be adopted because it answers more of your real questions at an acceptable cost, and this comparison is the way to find out. Include simple questions in the set as well, to check that an agent does not make them slower or worse.

A sensible way to adopt it

Start with a fixed pipeline and a test set, then look at which questions fail and why. If a particular group needs a new capability, add the smallest decision that deals with it. That might be one router, a single retry or a split step for comparison questions. Keep the rest fixed.

As decisions accumulate, branching and retry logic can become difficult to manage in plain functions. At that point, a graph library can make the state and routes explicit, as shown in building an orchestrator with LangGraph. A fully autonomous agent with many tools should be the last step, justified by evidence, and not the starting point.

If you are weighing up this design for a real system, the RAG development page describes what I can help with.

Frequently asked questions

What is the difference between RAG and agentic RAG?

Plain RAG follows a fixed path of retrieving and then generating. Agentic RAG lets a model make decisions along the way, such as whether to search, which source to use and whether to search again.

Is agentic RAG always more accurate?

No. It can help with multi-step questions, but it adds new failure points and will not cure poor parsing or an inadequate index. Compare it against a fixed pipeline on your own test set.

Do I need a framework to build agentic RAG?

No. A simple loop or a single router can be written in ordinary code. A graph library becomes more useful as state, branching, retries and pauses for review become harder to manage by hand.

How do I stop an agentic RAG system from looping forever?

Enforce a maximum number of searches or steps in code, rather than asking the model to stop. When the limit is reached, return a clear statement that the system could not find enough information.

Agentic RAG explained: when it beats a plain pipeline