SQL agents vs RAG: choosing by the shape of your data

The core difference

The two approaches do different kinds of work. A SQL agent translates a question into a database query, runs it and explains the result. The database does the reasoning, so the figure comes from a calculation over the relevant rows. RAG searches for passages that resemble the question and asks a model to compose an answer from them, so the answer depends on what a handful of retrieved passages say.

A small example shows why this matters. Take the question "What was our total refund value last quarter?" Suppose you embed each order row as text and use RAG. The search returns perhaps twenty similar-looking rows, and the model adds up what it was given. The result is a confident total based on a tiny sample of the data, and nothing in the answer reveals that. A SQL query covers every matching row.

The reverse fails too. "What is our refund policy for damaged goods?" has no meaningful answer in a table of orders, and storing long documents in a text column and matching keywords with LIKE gives you a poor imitation of search.

A side-by-side comparison

Sort your real questions first

Before choosing, collect a sample of the questions people actually ask, perhaps from support logs or by asking users directly, and label each one. Three labels are usually enough: answered by computing over rows, answered by reading text, or needing both. The proportions often settle the matter, and they will also show you questions that neither approach can answer because the data does not exist.

Some phrases give useful hints, although they are not rules:

• Likely SQL: "total", "average", "how many", "top ten", "compared with last month", "list all customers who".
• Likely RAG: "what does the contract say", "how do I", "explain", "what is the process for".
• Likely both: "which of our delayed orders qualify for compensation under the policy?"

Watch the word "all" in particular. "List every contract that mentions a termination fee" looks like a document question, but it demands complete coverage, which top-matching retrieval does not offer. It may be better to extract those fields from each contract into a table in advance, then query the table.

When a SQL agent is the better choice

• Answers require aggregation or complete coverage. Sums, counts, groupings and exhaustive lists all rely on seeing every row.
• The data is live and already well modelled. Clear tables, keys and consistent values make queries much easier to generate correctly.
• Users want to explore. Business questions vary too much for a fixed set of reports, and follow-up questions are natural.

The catch is that a query can run perfectly and still be wrong. If "active customer" has a specific meaning in your business, and the model guesses another, the number will look entirely reasonable. Give the agent definitions and a curated set of views, and consider a semantic layer where metrics have one approved definition. Then treat the database as a security boundary. Text-to-SQL agents: risks and safeguards covers read-only roles, query validation and resource limits in detail, and those safeguards are not optional.

Also ask whether you need an agent at all. If most questions fit a handful of known shapes, parameterised queries exposed as narrow tools are safer and easier to test than free-form SQL generation.

When RAG is the better choice

• The knowledge is written down in prose. Policies, manuals, wikis and contracts are the natural sources.
• The answer is contained in a few passages. Retrieval works when a small amount of text can resolve the question.
• Users need to check the source. Citations let them read the original wording, which matters for policy or legal text.

RAG's failures tend to come from the content and the search step, and the guide to what RAG is and when to use it walks through them. Its measures also differ from a SQL agent's, which is covered in how to evaluate a RAG system.

Awkward data in between

Real data rarely divides so tidily. A few cases come up often:

• Free text inside a database. Tickets and product descriptions have both columns and prose. Filter on the columns, then search the text. Metadata filters in a retriever handle simple cases, while a database with vector support, as discussed in pgvector vs dedicated vector databases, can combine both in one query.
• Tables inside documents. A PDF price table is structured data in an unstructured container. RAG for PDFs and tables describes the options, including extracting tables to a store you can query if people need to compute over them.
• Spreadsheets. A single lookup may be fine as text, but any calculation should be done by loading the sheet into a queryable form.
• Numbers quoted in reports. "What did the annual report say about revenue?" is a RAG question about a statement. "What was revenue?" may be a database question. The two can disagree, so decide which source is authoritative.

Combining the two with a router

When questions fall on both sides, a common design exposes each capability separately and chooses between them per question. The simplest form makes one classification call and then runs a fixed path. It is illustrative, with placeholder functions:

# Illustrative only. All helper functions are placeholders.
ROUTES = {"data", "documents", "both", "unclear"}

def answer(question, user):
    route = classify_question(question)
    if route not in ROUTES:
        route = "unclear"

    if route == "unclear":
        return ask_for_clarification(question)

    context = {}
    if route in {"data", "both"}:
        context["rows"] = run_safe_query(question, user)
    if route in {"documents", "both"}:
        context["passages"] = retrieve_passages(question, user)

    return write_answer(question, context)

Note that user comes from your application and goes to both paths, so that permissions apply on each side. A wrong route gives a wrong answer however good the later steps are, so add routing to your test set as a score of its own, and allow an "unclear" outcome rather than forcing a guess.

Some questions need steps that depend on one another, for instance reading a policy to find a threshold and then querying for orders over it. A fixed router cannot do this, whereas an agent that chooses tools in a loop can, at the cost of more calls and more ways to go wrong. Agentic RAG explains that trade-off. Begin with the fixed path and move to a loop only for questions you have seen it fail on.

Testing each path

Test the two paths differently. For SQL, write questions with known answers checked by hand, and compare results rather than query text, because different queries can be equally correct. For RAG, score retrieval and the final answer separately. For both, include questions that should produce "I cannot answer that from the available data".

Show your evidence in the interface too. A generated query with its row count, or a citation beside a claim, lets people catch mistakes that an automated test missed.

If you are deciding how to approach a mix of documents and databases, the RAG development page describes what I can help with.

Frequently asked questions

Can RAG answer questions about data in a database?

It can handle lookups of individual records converted to text, but it is unreliable for totals, counts and exhaustive lists because retrieval returns only a few matching items. Use a database query whenever the answer depends on considering all the relevant rows.

Is a SQL agent more accurate than RAG?

Neither is more accurate in general, because they answer different kinds of question. A database calculates exactly what the query asks, but the query may misinterpret the question. RAG can be accurate about written content, but it depends on finding the right passages.

Can I use a SQL agent and RAG together?

Yes. A router can send each question to the database, the documents or both, and then an LLM can write the answer from the results. Apply the user's permissions on both paths, and test the routing step separately.

Do I need a vector database for a SQL agent?

Generally not, because a SQL agent queries your existing tables. Embeddings can still help with finding relevant tables in a large schema, or matching similar past questions to example queries, but they are an optional aid rather than a requirement.