What Is RAG and When Should You Use It?

The problem RAG solves

A large language model knows only what was represented in its training data, and it has no built-in access to your internal documents. Ask it about your refund policy and it may either decline or produce a plausible answer that happens to be wrong. Since models generate likely text rather than look up facts, a confident tone is no guarantee of accuracy.

The most direct fix is to give the model the relevant information along with the question. Retrieval-augmented generation, or RAG, automates this. Rather than expecting the model to remember your content, the system finds the relevant passages at question time and places them in the prompt.

How a RAG system works

Most RAG systems consist of two pipelines. One runs ahead of time to prepare the content, and the other runs each time a question arrives.

The ingestion pipeline

Load and parse. Documents are pulled from their sources and converted to clean text. PDFs, tables and scanned pages can all lose structure at this step.
Chunk. Long documents are split into smaller passages, so that retrieval can return just the relevant part. The guide on choosing chunking strategies covers this decision.
Embed. An embedding model turns each chunk into a vector, a list of numbers positioned so that texts with similar meanings tend to sit close together.
Index. Vectors, text and metadata such as source, title and date are stored in a searchable index.

The query pipeline

Process the question. The question is embedded with the same model. In a conversation, a follow-up such as "what about the second one?" may first need rewriting into a standalone question.
Retrieve. The system finds the chunks closest to the query, often combining vector search with keyword search. See hybrid search and reranking for RAG.
Filter and rerank. Metadata and permission filters, ideally applied during the search itself, exclude passages that are out of date or out of scope. Optionally, a reranking model then reorders the candidates.
Generate. The chosen passages go to the LLM with instructions to answer from them, cite them and say so when they do not contain the answer.

In simplified form, the query path looks like this:

def answer(question):
    candidates = search_index(question, limit=20)
    passages = rerank(question, candidates)[:5]

    if not passages:
        return "No relevant source was found."

    prompt = build_prompt(question, passages)
    return call_llm(prompt)

The function names are placeholders and the limits are illustrative; suitable values depend on your content and should be tested. Search usually returns something even for an off-topic question, so the empty check catches little on its own. The prompt must also tell the model to decline when the passages do not answer the question.

For how vectors work and what to look for in a store, read embeddings and choosing a vector database.

When RAG is a good fit

RAG tends to work well when several of these conditions apply:

The answer exists in your content. Documentation, policies, knowledge bases and manuals are typical sources.
The content changes. Updating an index is far simpler than changing a model.
The content is too large for one prompt. Retrieval selects a small relevant portion instead of sending everything.
Users need to verify answers. Because each answer is linked to specific passages, citations can be shown.
Access to the content is controlled. Retrieval can be filtered by what each user is allowed to see, provided the filter is enforced outside the model.

When RAG is the wrong tool

RAG is popular enough that it is sometimes applied by default. Several situations call for something else.

Questions about structured data. "What was the total for each region?" requires a calculation over all the relevant rows, which similarity search will not provide. A database query, possibly generated with an LLM and carefully constrained, is a better match.
A small, stable knowledge base. If everything fits comfortably in the prompt, including it directly avoids a whole retrieval system.
Style or behaviour problems. If the issue is how the model writes rather than what it knows, better instructions, examples or fine-tuning are more relevant.
Questions requiring whole-corpus reasoning. "What are the main themes across all of these reports?" is poorly served by a handful of top-matching chunks. A map-and-summarise approach or precomputed summaries may do better.
Real-time facts. Live values such as stock levels are often better fetched from an API through a tool call than from an index that may lag behind.

RAG compared with the alternatives

Three approaches are often confused, and they solve different problems:

Prompt context places known information directly in the prompt. It is the simplest option, but limited by the size of the model's context window and by cost.
RAG selects information dynamically for each question. It is suited to large or changing knowledge, but adds a pipeline to build and maintain.
Fine-tuning changes the model's behaviour through training examples. It can help with consistent format or tone, but it is not a reliable way to keep facts current.

These are not mutually exclusive. A system can use a fine-tuned model with retrieved context and a fixed set of instructions in the prompt.

Where RAG systems commonly go wrong

When a RAG answer is poor, the cause is often located before the model is ever called. Common sources of trouble include:

Poor parsing. Broken tables and stray headers produce chunks that cannot be retrieved or understood.
Chunks without context. A passage reading "this limit applies to both plans" means little without the section it came from.
Vocabulary mismatch. Users' terms may differ from those in the documents. Query rewriting and hybrid search can help.
Exact-match failures. Vector search alone may miss a specific error code or part number, which is why keyword search is often added.
Stale or conflicting sources. Without metadata and a process for removing old content, the system may cite an outdated policy.
Prompt injection through documents. Retrieved text is untrusted input. If it contains instructions, the model may try to follow them, so system permissions should not depend on the model ignoring them. See LLM guardrails and prompt injection defence.

Checking that it works

Before relying on a RAG system, test it on realistic questions, including some the content cannot answer, and score retrieval and answers separately, because they fail in different ways. The guide on how to evaluate a RAG system covers the test set, the measures and how to act on the results.

Where to start

Begin with a representative subset of your documents and a basic pipeline, along with your question set. Then inspect what is retrieved for failing questions before adding anything sophisticated. Reranking, query rewriting and more complex chunking are all useful, but each should be added in response to a problem you have observed.

If you are planning a system of this kind, the RAG development page describes what I can help with.

Frequently asked questions

What does RAG stand for?

RAG stands for retrieval-augmented generation. A system retrieves relevant information from a knowledge source and supplies it to a language model, which uses it to generate an answer.

Does RAG eliminate hallucinations?

No. It can reduce them by grounding answers in source material, but a model can still misread or go beyond a passage, and retrieval can return the wrong content. Citations and evaluation help to manage this risk.

Is a vector database required for RAG?

No. Retrieval can use keyword search, a general database with vector support or a dedicated vector store. RAG describes the pattern of retrieving before generating, not one specific technology.

How is RAG different from search?

Search returns documents or passages for a person to read. RAG uses search as one step, then has a language model compose an answer from the results. Its quality therefore depends heavily on the search step.

What RAG is and when to use it