NER and Data Extraction with LLMs

NER and extraction are related but different

Named entity recognition, or NER, finds mentions of things in text and labels their type: people, organisations, places, dates, amounts. Its output is a set of spans, each pointing to a position in the source text.

Structured data extraction asks for more. It fills a defined record from a document, such as the supplier, invoice date, total and line items of an invoice. Values may need normalising, inferring from context or combining from several places. A date written in words becomes a standard format, and "the Supplier" in a contract has to be resolved to the company named in its opening paragraph.

The distinction matters because the two tasks fail differently. NER can be scored by checking spans against the text. Extraction also has to be right about meaning, relationships and which of several candidates is the correct one. Both are common NLP development tasks, and if you only need to assign a label to a whole text, text classification is the simpler problem.

When a traditional tool is the better starting point

LLMs are not always needed. Consider these options first:

Patterns and validators suit values with a fixed format, such as email addresses, reference numbers or postcodes. They are cheap and deterministic, and a checksum or format rule can also confirm the result.
Pretrained NER models from open-source NLP libraries cover common entity types and run locally at low cost. Quality depends on how closely your text resembles what they were trained on.
Custom trained models suit domain-specific entities when labelled examples are available and the task is stable.
Layout-based extraction suits forms and templates whose fields always appear in the same place.

LLMs earn their place where documents vary in layout and wording, where a value must be understood from context, or where the schema changes too often for retraining to be practical. Mixing methods is normal: a pattern can capture a reference number while an LLM interprets a free-text clause in the same document.

Design the schema before the prompt

The schema is the contract between the model and the rest of your system, and many extraction problems are really schema problems.

Make absence explicit. Optional fields should allow null, and instructions should say to use it when a value is not stated. If every field is required, the model is under pressure to invent something.
Use constrained types. Prefer enumerations, numbers and standard date formats to free text where the values are known.
Define every field in words. "Date" is ambiguous on an invoice that has an issue date and a due date. Name them separately and describe each one.
Specify formats and units. Say how to represent currency, and keep the amount and currency in separate fields, so that nothing relies on the model converting values.
Keep nesting modest. Deeply nested schemas are harder to review and more likely to go wrong. Consider several smaller passes for a complex document.
Ask for evidence. A field for the exact supporting quote gives you something to check and something to show a reviewer.

# Illustrative only. A schema expressed as Python dataclasses.
from dataclasses import dataclass
from typing import Optional

@dataclass
class ExtractedValue:
    value: Optional[str]    # None when the text does not state it
    evidence: Optional[str] # exact quote supporting the value

@dataclass
class Invoice:
    supplier_name: ExtractedValue
    issue_date: ExtractedValue  # ISO 8601 date, for example 2026-01-31
    due_date: ExtractedValue
    total_amount: ExtractedValue
    currency: ExtractedValue    # three-letter currency code

Getting structured output from the model

Many LLM providers offer a way of requesting JSON or of constraining output to a schema, and tool calling can be used for this as well. The details differ between providers and change over time, so check the current documentation for yours. Where available, these features reduce formatting errors considerably.

They do not make the content correct. A response can match the schema perfectly and still contain the wrong date. So treat validation as a series of layers:

Syntax. Does it parse?
Schema. Are the fields and types right?
Business rules. Does the due date fall after the issue date? Do the line items add up to the total?
Grounding. Does the value, or the quote offered to support it, appear in the source document? A normalised value such as a reformatted date may not appear literally.

If syntax or schema validation fails, one retry that includes the specific error often resolves it. If a value breaks a business rule, send it for review, as repeatedly asking the model may only produce a different guess.

Guarding against invented values

The characteristic failure of LLM extraction is a value that looks plausible but is not in the document. Traditional NER rarely fails this way, since it can only label text that exists. The evidence field makes a simple check possible:

# Illustrative only. Checks that a quote exists in the source.
import re

def normalise(text):
    return re.sub(r"\s+", " ", text).strip().lower()

def is_grounded(evidence, source_text):
    if not evidence:
        return False
    return normalise(evidence) in normalise(source_text)

A passing check shows that the quote exists, not that it supports the value or that the value was interpreted correctly, so this is a filter and not a guarantee. Quotes may also fail for innocent reasons, such as line breaks or errors from optical character recognition, so a failure should generally route the record to review and not simply discard it.

Prompts help as well: tell the model to use only the document, to return null when a value is not stated and not to apply outside knowledge. Provide examples in which null is the correct answer, because examples where every field is filled teach the opposite habit. Bear in mind that document text is untrusted input, and a document could contain instructions aimed at the model. The guide on LLM guardrails and prompt injection defence covers how to limit that risk.

Long documents and repeated entities

A long document may not fit in the context window, and even when it does, attention to detail can suffer. Common approaches include:

Find the section first. Locate the relevant part with search or headings, then extract from that part. The methods overlap with chunking for RAG, although here a split entity or table is the main thing to avoid.
Extract per chunk and merge. Process sections separately and combine the results. You will need rules for duplicates and for conflicts between sections.
Run field-specific passes. Ask for fewer fields per call, in exchange for more calls.

Merging raises the problem of entity resolution: deciding whether "Acme Ltd", "Acme" and "the Company" refer to the same organisation. Decide this in a defined step, using a reference list of known entities where you have one. A code-based match against known records is generally easier to audit than leaving it to the model.

Check the input as well. PDFs and scans often lose reading order or scramble tables during text extraction, and no prompt can recover information that the parser has destroyed.

Measuring extraction quality

Create a labelled set from real documents and include awkward ones: unusual layouts, poor scans, documents in which a field is truly missing. Then score at the field level, because an overall average hides which fields are unreliable.

Span matching for NER. Decide in advance whether a partial match counts. Exact matching is stricter, while partial matching may be more realistic for some uses. Report which one you used.
Value matching for extraction. Normalise before comparing, so that different formats of the same date count as equal.
Precision and recall per field. Precision shows how often an extracted value is right. Recall shows how often a present value is found.
Correct nulls. Track separately how often the model correctly returned nothing, because invented values are often the most damaging errors.
Whole-record accuracy, if a downstream process needs every field in a record to be right.

Putting extraction into production

Route by risk. Send records that fail validation, or fields where errors are costly, to a reviewer. The source quote beside the value makes this quicker.
Record provenance. Store the document version, the prompt and schema versions, the model and the evidence for each value, so that a bad result can be traced.
Feed corrections back. Reviewer changes are valuable test cases and can show which fields need better definitions.
Take care with personal data. Extraction often deals with names and contact details. Check where text is sent and retained, and how long logs keep it.
Version the schema. Downstream consumers will depend on it, so changes need the same care as any other API change.

Frequently asked questions

What is the difference between NER and data extraction?

Named entity recognition finds and labels mentions in text, such as people or dates. Structured data extraction fills a defined record, which can involve normalising values, using context and working out relationships between them.

Does structured output guarantee correct data?

No. Schema-constrained output can reduce formatting errors, but a well-formed response can still hold a wrong value. You still need business rule checks, grounding checks and evaluation against labelled documents.

How do I stop an LLM inventing values?

You cannot rule it out, but you can reduce and catch it. Allow nulls, tell the model to use only the document, include examples where null is correct, ask for a supporting quote and verify that the quote exists in the source.

Should I still use a traditional NER library?

Often, yes. Pretrained models and patterns can be cheap, fast and able to run locally, which suits common entity types and fixed formats. An LLM is more attractive when layouts vary or a value depends on context.

Named entity recognition and data extraction with LLMs