Structured output from LLMs: getting reliable JSON

The four levels of enforcement

"JSON mode" and "structured outputs" are often used interchangeably, but the options differ in what they actually guarantee. From weakest to strongest:

Avoid the older workaround of defining a fake tool just to obtain structure if a direct format option exists. Forcing a tool is also not universally supported: Anthropic's documentation, for example, lists models where forced tool choice returns an error and points to strict tools or structured outputs instead.

How the main providers expose it

Parameter names change, so treat the following as a snapshot from each provider's documentation in November 2026, and check the current pages before you build:

• OpenAI. The Responses API accepts a json_schema format with "strict": true under text.format. Its Python SDK has responses.parse, which takes a Pydantic model as text_format and returns output_parsed.
• Anthropic. The Messages API takes a schema in output_config.format, and strict: true on a tool enforces its input_schema. The Python SDK offers client.messages.parse() with a Pydantic model. The older output_format parameter is marked as deprecated.
• Google Gemini. The documentation describes a response_format object with a mime_type of application/json and a schema. In Python you can pass a Pydantic model's model_json_schema().

Know what the schema will not enforce

Each provider supports only part of JSON Schema, and the unsupported parts behave differently. Anthropic's documentation says that numeric limits such as minimum and string length limits are unsupported and return a 400 error, though its SDK helpers can move some of them into descriptions and validate locally. Gemini's documentation says minLength, maxLength and uniqueItems, among others, are silently stripped. A rule you believed was enforced may therefore not be, which is one reason to validate on your side regardless.

Very large or deeply nested schemas can also be rejected, and providers publish complexity limits. Schema compilation can add latency on first use as well. Keep schemas small, and split a sprawling one into separate calls if the parts are really separate tasks. Smaller schemas are also easier to review when a field starts misbehaving.

Three ways valid-looking output still fails

Schema enforcement does not cover these situations, so your code must handle each of them.

• Refusals. The model may decline for safety reasons, and the response may not follow your schema. OpenAI exposes a refusal field or content item, and Anthropic uses a stop_reason of refusal.
• Truncation. If the output hits the token limit, the JSON can be cut off midway. OpenAI's Responses API marks this with a status of incomplete and a reason of max_output_tokens, while Anthropic returns a stop_reason of max_tokens. Increase the limit or ask for less, since retrying the same request may simply fail the same way.
• Correct shape, wrong content. A date can be valid and still be the wrong date. Google's documentation makes this point directly: schema-compliant output can still be semantically wrong.

Check the status before parsing. A parse error on a truncated reply tells you little, while "truncated at the token limit" tells you what to change.

Design the schema for the model

A good schema reduces mistakes before validation ever runs.

• Use enums for closed sets. Add an other or unknown value, or the model has to force every case into a category that does not fit. Anthropic's documentation warns that enum capitalisation can vary, so normalise before comparing.
• Make absence explicit. A required but nullable field gives the model a clear way to say "not stated". A model forced to fill in a required string may invent one.
• Describe fields precisely. Field names and descriptions guide the model. State formats and units, such as an ISO 8601 date or an amount in pence.
• Keep it flat. Deep nesting adds complexity and more ways to go wrong.
• Put a short justification first, if you want one. Fields tend to be generated in order, so a brief evidence field placed before a verdict gives the model a chance to ground it. Keep it short, and ask for an explanation or supporting quote. Anthropic warns that asking for step-by-step reasoning in a property can trigger a refusal.

For extraction work in particular, where the concern is invented values and field-level accuracy, see named entity recognition and data extraction with LLMs.

Validate in your own code

Provider guarantees cover part of the problem, can vary by model and may change. Your application should have its own definition of valid data. Pydantic suits this in Python, because one model can provide the JSON Schema for the request and the validation for the response, plus rules that a schema cannot express. The code below is illustrative, using a made-up invoice example.

# Illustrative only. Requires Pydantic v2.
from datetime import date
from typing import Literal

from pydantic import BaseModel, ConfigDict, ValidationError, model_validator


class Invoice(BaseModel):
    model_config = ConfigDict(extra="forbid")

    supplier: str
    issue_date: date | None
    due_date: date | None
    currency: Literal["GBP", "EUR", "USD", "INR", "other"]
    total_minor_units: int | None

    @model_validator(mode="after")
    def check_rules(self):
        if self.issue_date and self.due_date and self.due_date < self.issue_date:
            raise ValueError("due_date is before issue_date")
        if self.total_minor_units is not None and self.total_minor_units < 0:
            raise ValueError("total_minor_units is negative")
        return self

Cross-field rules like these are exactly what schema enforcement does not catch. For business rules that depend on other data, such as whether a supplier exists in your system, run a further check against your database after parsing.

Retry with the error, and only once or twice

When validation fails, sending the error back gives the model something specific to fix. Keep the number of attempts low, because each one costs tokens and time, and repeated failure usually points to a bad schema or an ambiguous input. The illustrative function below handles refusals and truncation separately, since neither should be treated as a repairable format problem. Here call_model is a placeholder for your own adapter.

# Illustrative only. call_model returns an object with .status and .text.
class OutputFailed(Exception):
    pass


def get_invoice(document, call_model, max_attempts=2):
    messages = [{"role": "user", "content": build_prompt(document)}]
    for _ in range(max_attempts):
        reply = call_model(messages, schema=Invoice.model_json_schema())
        if reply.status == "refused":
            raise OutputFailed("refused")
        if reply.status == "truncated":
            raise OutputFailed("hit the token limit; shorten input or raise it")
        try:
            return Invoice.model_validate_json(reply.text)
        except ValidationError as exc:
            messages.append({"role": "assistant", "content": reply.text})
            messages.append({
                "role": "user",
                "content": f"That output failed validation: {exc}. "
                           "Return corrected JSON only.",
            })
    raise OutputFailed("validation failed after retries")


def build_prompt(document):
    return (
        "Extract the invoice fields from the document below. "
        "Use null for anything not stated. Do not guess.\n\n"
        "<document>\n" + document + "\n</document>"
    )

Decide what happens after the final failure. Sending the item to a review queue or returning a clear error is almost always better than saving a best guess. Be careful with validation messages too, as they can echo input values into logs, which matters if your documents contain personal data.

Treat output as untrusted

Valid JSON can still carry harmful values. If a document contains injected instructions, the model may place attacker-chosen text in perfectly valid fields. Use parameterised queries, escape values before showing them in a page and never pass a field straight to a shell or an eval. LLM guardrails and prompt injection defence explains the wider approach.

Streaming and schema changes

Partial JSON is not valid JSON, so a streamed response cannot be fully validated until it ends. If you stream into an interface, use a tolerant partial parser for display only, and validate the complete object before acting on it. Streaming LLM responses to a web UI covers the transport side.

Treat a schema as a versioned contract as well. Record the schema version with each stored result and re-run your tests whenever you change a field, since renaming one can change what the model produces.

Test more than the format

Once the format is enforced, your failure rate for parsing may look excellent while accuracy remains unknown. Build a small labelled set and measure correctness per field, with a separate look at how often the model returns null when a value was present, or a value when none was. Include inputs that lack the information altogether, along with empty or very long documents, and log refusal, truncation and retry rates in production so that you can see changes over time.

If you are adding this kind of feature to a product, the generative AI and LLM development page describes what I can help with.

Frequently asked questions

What is the difference between JSON mode and structured outputs?

JSON mode aims for syntactically valid JSON without enforcing any particular shape. Structured outputs also constrain the response to a schema you supply. OpenAI's documentation draws this distinction and recommends structured outputs where they are available.

Do structured outputs guarantee correct data?

No. They address the shape of the output and not whether the values are true. A model can return a schema-valid date that is wrong, so validate business rules in your code and measure accuracy on labelled examples.

Should I still validate if the provider enforces my schema?

Yes. Refusals and truncation can produce output that does not match, some schema constraints may not be enforced, and cross-field rules cannot be expressed. Your own validation also protects you if you change model or provider.

Should I use a tool call or a response format to get JSON?

Use a response format when the JSON is the final answer, for example extracted data. Use a tool call when the model is choosing an action your application will carry out. Some requests legitimately combine both.