Document summarisation with LLMs

Decide what the summary is for

A summary is not one fixed output. An executive wants the decisions and risks in a report, a lawyer may want the obligations in a contract, and a support lead may want recurring complaints in a set of tickets. Each of these keeps different information, so a general "summarise this" instruction leaves the most important choice to the model.

Before choosing a method, write down the following:

• The reader and their next action. What should this person be able to decide or do afterwards?
• What must not be lost. Dates, amounts, named parties, exceptions and caveats are often the first things to disappear.
• The format. Decide on paragraphs, bullet points or fixed fields, and on a target length.
• The cost of an error. A rough digest of an internal meeting can tolerate more than a summary of a medical or legal document, where review by a qualified person may be needed.

These answers often change the design. A request for fixed fields, for instance, may really be an extraction task, covered in the guide to entity and data extraction with LLMs.

Extractive and abstractive summaries

Traditional NLP distinguishes two broad kinds of summary. An extractive summary selects sentences from the source unchanged. An abstractive summary writes new sentences, and this is what LLMs generally do. Extractive output cannot misquote the source, although selected sentences may read awkwardly or mislead without their context. Abstractive output reads better but can introduce errors. The two can also be combined, as the extract-first approach below shows.

Approach 1: send the whole document

If the document fits comfortably within the model's context window, this is the simplest choice and a sensible baseline. The model sees all of the text at once, so it can connect a claim in one section with a qualification in another.

The limits are cost, latency and attention. A larger input costs more, and a document that technically fits may still give uneven coverage. Check whether material from the middle is underrepresented, and do not assume that an advertised context limit is a practical target. Check your provider's current limits rather than relying on figures from an article. Remember also to leave room for the output.

Approach 2: map-reduce

Map-reduce splits the document into chunks, summarises each one separately (map) and then combines those summaries (reduce). Here is the basic shape. This sample is illustrative: call_llm and split_by_section are placeholders for your own functions, and error handling is left out.

from concurrent.futures import ThreadPoolExecutor

def summarise_section(section):
    return call_llm(
        "Summarise this section. Keep names, dates, amounts and "
        "exceptions. Use only the text provided.\n\n" + section
    )

def map_reduce_summary(document):
    sections = split_by_section(document)

    with ThreadPoolExecutor(max_workers=4) as pool:
        partials = list(pool.map(summarise_section, sections))

    combined = "\n\n".join(partials)
    return call_llm(
        "Combine these section summaries into one summary. Remove "
        "repetition and do not add new facts.\n\n" + combined
    )

• Strengths: it handles long documents, runs in parallel and covers each section.
• Weaknesses: each chunk is processed without knowledge of the others, so a pronoun or a term defined earlier can lose its meaning. The combining step may also discard details.
• Suits: collections of separate documents, or long texts with independent sections, such as reports and sets of reviews.

If the combined summaries are themselves too long for one call, group them and repeat the process. Each additional level loses some information, which leads to the hierarchical approach.

Approach 3: refine

Refine processes chunks in order and maintains a running summary. Each call receives the current summary and the next chunk, then returns an updated summary.

• Strengths: it retains the order of events and carries earlier context forward, which suits narratives, transcripts and arguments that build over time.
• Weaknesses: it runs one step at a time, so it can be slower. An early error may also persist, and the running summary can give more weight to earlier or later material, depending on how it is updated.
• Suits: sequential documents where each part depends on what came before.

Approach 4: hierarchical summaries

A hierarchical method follows the document's own structure. Subsections are summarised, then sections, then the entire document. It is similar to multi-level map-reduce, but its boundaries come from the author's organisation and not an arbitrary length, so each step works on a coherent unit.

A further benefit is that the intermediate summaries are useful in their own right. A reader can start with the overview and drill into a chapter, and each level can link to its source. They can also be indexed for retrieval, which helps with broad questions that a handful of matching chunks would answer poorly. What RAG is and when to use it explains this limitation of ordinary retrieval.

Approach 5: extract first, then write

Sometimes it helps to separate finding from writing. In the first pass, the model pulls out specified items, such as decisions, obligations or action points, with a reference to where each was found. In the second, it writes the summary using only those items.

The intermediate list gives you something to verify, as you can check in code that a quotation or a figure really appears in the source. The drawback is that anything missed by the first pass cannot reach the summary. This approach suits contracts, meeting notes and other material where the reader knows what they are looking for.

Choosing between the approaches

• The document fits comfortably: start with the whole document.
• The sections are largely independent, or there are many documents: try map-reduce.
• The order matters and later sections rely on earlier ones: try refine.
• The document is long with clear headings, or readers need to drill down: try hierarchical.
• The reader needs specific items, and accuracy is critical: try extract first.
• The reader cares about only one topic: retrieve the relevant passages and summarise those.

Chunking affects most of these. Split along headings where possible, keep the section title with each chunk and carry defined terms forward where necessary. Parsing matters too, because a broken table or lost heading will distort what the model reads. See choosing chunking strategies; although written for RAG, its discussion of boundaries applies here as well.

Controlling length and cost

Models are not precise about length, so a request for exactly 100 words may produce something close but not exact. Ask for a limit on sentences or bullet points where that suffices, and check it in code if a hard limit matters. Section-based methods also multiply the number of calls, so estimate the total before processing a large collection. A smaller, cheaper model may be adequate for the map step, with a stronger one used to combine the results, but test this rather than assuming it. Store section summaries against a hash of the source text, so that an updated document only needs its changed sections processed again.

Checking that a summary can be trusted

Summaries are hard to assess because a poor one can read perfectly well. Look at several qualities separately:

• Faithfulness. Is each statement supported by the source? Watch for merged facts, changed numbers and qualifications that have been dropped, because "may" easily becomes "will".
• Coverage. Are the points this reader needs included? List them for each test document and count how many appear.
• Usefulness. Can the reader act on it? This is best answered by the reader.

Some practical controls help. Ask for section or page references next to claims, and show them to readers. Check in code that numbers, dates and names in the summary also appear in the source; this will not catch everything, but it is cheap. An LLM judge can check claims against the source, provided you have compared its verdicts with human judgement. Build a small test set with reference points written by someone who knows the material.

Long documents can also contain text that tries to steer the model, and a summary may be read as authoritative. Treat the document as untrusted input, particularly if the summary feeds an automated action. The guide on guardrails and prompt injection covers this risk.

Where to start

Take a few representative documents and write down what a good summary must contain. Compare a whole-document baseline with one section-based method, measuring faithfulness, coverage, cost and latency. Choose the simplest method that meets your needs. If you are planning a feature of this kind, the NLP development page describes what I can help with.

Frequently asked questions

What is the difference between map-reduce and refine?

Map-reduce summarises each chunk independently, then combines those summaries, so the chunks can be processed in parallel. Refine works through chunks in order and updates one running summary, which keeps sequence but must run one step at a time.

Should I use RAG to summarise a document?

Usually not for a summary of the whole document, because retrieval returns only the passages closest to a query. RAG can suit a targeted summary on one topic, such as what a contract says about termination.

How do I stop a summary inventing details?

You cannot rule it out, but you can reduce and detect it. Instruct the model to use only the source, ask for references to sections, verify names and figures in code and test faithfulness on a sample.

Is a bigger context window enough for long documents?

Not always. A document that fits may still exceed what is practical on cost or latency, and details from the middle can be underrepresented. Compare a whole-document summary with a section-based one on your own material.