Choosing Chunking Strategies for RAG

Why chunking matters

In a RAG system, documents are divided into chunks before they are indexed. Retrieval returns chunks rather than whole documents, so a chunk is the basic unit that can be found and shown to the model. If the answer is split across two chunks, or buried in a chunk about something else, even a strong model may fail.

Chunk size involves a real tension:

Chunks that are too large mix several topics. Their embeddings represent a blend of those topics, which can make them harder to match with a specific question, and they use more of the model's context on irrelevant text.
Chunks that are too small match precisely but can lose the surrounding meaning. A sentence such as "it must be renewed annually" is of little use if nothing says what "it" is.

Ideally, a chunk should make sense on its own and cover one idea. Different strategies approach that goal in different ways.

Before choosing: look at your data

Good chunking depends on good parsing. If headings have vanished, tables have collapsed into jumbled text or page footers run through every paragraph, no splitting method will rescue the result. Inspect the extracted text from a sample of documents before tuning anything else.

Also consider what your questions look like. Short factual lookups favour focused chunks, whereas questions that require explanation may need more surrounding text. A knowledge base of short articles and a set of long contracts are likely to need different treatment.

The main strategies

Fixed-size chunking

This method splits text every N tokens or characters, usually with some overlap between neighbouring chunks.

Strengths: simple, fast, predictable in size and a useful baseline.
Weaknesses: ignores meaning and structure, so it can cut through sentences, lists and tables.
Suits: uniform prose, early prototypes and comparison with other methods.

Recursive chunking

Recursive splitting tries natural separators in order: first paragraph breaks, then line breaks, then sentences, then words. It stops dividing once a piece fits within the size limit.

SEPARATORS = ["\n\n", "\n", ". ", " "]

def split(text, max_size, seps=SEPARATORS):
    if size(text) <= max_size or not seps:
        return [text]

    chunks = []
    for part in text.split(seps[0]):
        if part.strip():  # skip empty pieces
            chunks += split(part, max_size, seps[1:])
    return chunks

This simplified sketch leaves out the usual final step of merging small neighbouring pieces back together up to the size limit, along with restoring the separators and adding overlap. Note also that a piece with no separators left could still exceed the limit, so production code needs a hard cut as a last resort.

Strengths: respects text boundaries while keeping sizes under control.
Weaknesses: sees no logical document structure beyond its separators.
Suits: general-purpose text. It is a reasonable default for many projects.

Structure-aware chunking

This approach splits along the document's own structure, such as Markdown or HTML headings, numbered legal clauses, code functions or the rows of a table. Oversized sections are then split further with a recursive method.

Strengths: chunks follow the author's own organisation, and heading paths provide useful metadata.
Weaknesses: depends on parsing quality and may require separate handling for each format.
Suits: technical documentation, policies, contracts, manuals and code.

Semantic chunking

Semantic chunking compares the embeddings of neighbouring sentences and starts a new chunk where similarity falls sharply, on the theory that the topic has changed.

Strengths: can find topic boundaries in text with little formatting.
Weaknesses: costs more at ingestion, needs threshold tuning and can produce uneven sizes. Its added complexity does not always yield better retrieval, so it should be tested against a simpler baseline.
Suits: long, unstructured text covering several topics, such as transcripts.

Parent-child chunking

Parent-child chunking, also called small-to-big retrieval, separates what is searched from what is shown to the model. Small child chunks are indexed for precise matching, each linked to a larger parent such as its section. When a child matches, its parent goes to the LLM.

Strengths: directly addresses the tension between precise matching and sufficient context.
Weaknesses: adds storage and pipeline complexity, plus logic to remove duplicates when several children share a parent.
Suits: long documents where answers depend on surrounding context.

Other decisions that affect results

Overlap

Overlap repeats some text between adjacent chunks, so that an idea falling on a boundary is less likely to be lost. It increases index size and can return near-duplicate results, however, and it is less valuable when chunks already follow clean structural boundaries. Treat it as a setting to test rather than a requirement.

Measuring size in tokens

Models and embedding services limit input in tokens rather than characters. Measuring in tokens, ideally using a tokeniser appropriate to the embedding model, helps prevent chunks from being rejected or silently truncated. Check the limits of the model you actually use.

Context and metadata

A chunk often makes more sense with a little added context. Common techniques include:

Storing the document title, section path, date and source with each chunk.
Prepending the title and heading path to the chunk's text before embedding it.
Generating a short description that places the chunk within its document, at an additional cost during ingestion.

Metadata also enables filtering and citations, both of which often matter as much as the chunking itself.

Tables, code and lists

Generic splitters handle these formats badly. Try to keep a code function intact, keep a list with its introductory sentence, and keep table rows with their column headers. A row of numbers with no headers has very little retrievable meaning.

A method for choosing

There is no universally best strategy or chunk size. Published recommendations are useful starting points, but they were not derived from your documents or your questions. An experiment can settle the matter:

Build a test set. Collect realistic questions and note which passages contain each answer. Mark expected sources by document and text rather than chunk ID, since IDs change whenever you rechunk.
Set a baseline. Index your documents using recursive chunking at a moderate size.
Measure retrieval alone. For each question, check whether the expected passage appears in the top results, and at what rank.
Read the failures. Examine the chunks returned for failed questions. Failures often point straight to the fix: broken parsing, split tables, missing headings or a vocabulary mismatch.
Change one thing at a time. Try a different size, add heading context or switch to a structure-aware or parent-child strategy, and compare against the baseline.
Evaluate final answers. Once retrieval has improved, confirm that the generated answers improved too, using a method such as the one in how to evaluate a RAG system. Include the cost of context and any latency changes in your decision.

When the problem is not chunking

Chunking is a popular thing to tune, but not every retrieval failure is caused by it. If exact terms are being missed, hybrid search may be the fix. If relevant results appear but rank low, consider a reranker. If users phrase questions differently from your documents, look at query rewriting. If the content does not exist or is out of date, no retrieval technique will produce a good answer. Reading the failures, as above, shows you which problem you actually have.

A default path

For a new project, start with clean parsing and recursive or structure-aware chunking, then attach the title and section metadata. Add parent-child retrieval if answers lack context, and try semantic chunking only where the text has little structure and testing shows a benefit. Re-index and re-evaluate whenever the approach or embedding model changes, as vectors from different models or settings should not be mixed in the same index.

If you would like help applying this to your own documents, see RAG development.

Frequently asked questions

What is the best chunk size for RAG?

No single size is best. It depends on your documents, your questions and your embedding model. Start with a moderate size and test larger and smaller options against a set of real questions.

How much overlap should chunks have?

There is no fixed rule. A modest overlap is a common starting point for fixed-size or recursive chunking, while structure-aware chunks may need little or none. Compare results with and without it on your data.

Is semantic chunking better than recursive chunking?

Not necessarily. It can help with poorly structured text, but it costs more at ingestion and adds settings to tune. Treat recursive chunking as the baseline and adopt semantic chunking only if your evaluation shows a clear gain.

Do I need to re-embed documents after changing the chunking?

Yes. Changing chunk boundaries creates new text, so the affected chunks must be embedded and indexed again. Keep the ingestion pipeline repeatable so that this stays easy.

Choosing chunking strategies for RAG