LLM Observability and Tracing in Production

Why ordinary monitoring is not enough

Standard application monitoring tells you whether a request succeeded and how long it took. An LLM application can return HTTP 200 quickly and still give a wrong answer, invent a policy or call the wrong tool. Several other things make these systems harder to watch:

Failure is often silent. Nothing throws an exception when an answer is simply wrong.
Behaviour is variable. The same input can produce different outputs, and agents can take a different path on each run.
Cost varies by request. One long conversation or looping agent can use far more tokens than a typical request.
Causes sit upstream. A poor answer may come from retrieval, a prompt change or a model update, not from the final call.
Inputs keep shifting. Users ask things your test set never anticipated.

The shape of a trace

A trace represents one request from start to finish, made up of nested spans, each a timed operation. For a typical assistant with retrieval and tools, it might look like this:

request: answer_question
  retrieve_documents
    embed_query
    vector_search
  chat (model call 1)
  tool: get_order_status
  chat (model call 2)
  validate_output

This outline is an illustrative example rather than a required structure. Its value is that a slow or wrong response can be attributed to a step: you can see that the search returned nothing useful, or that the agent called the same tool three times. Give every step in a request a shared trace ID, and carry a conversation or session ID as well, so that related requests can be viewed together.

What to record on each step

Model calls

Model identity. Both the model you requested and the one the response reports, since aliases can resolve to a new version.
Tokens. Input and output counts, plus cached tokens if your provider reports them.
Timing. Total duration and, for streaming, the time to first token, which is often what users notice.
Finish reason. A sudden rise in length-limited responses or refusals is an early warning.
Settings and versions. Temperature, maximum tokens, and a version identifier for the prompt.
Errors and retries. Rate limits and timeouts, including how many attempts were made.

Retrieval steps

Record the query as sent, including any rewriting, the document and chunk IDs returned, their scores, filters applied and the index version. Without this, a bad answer cannot be traced back to the passages that shaped it. The guide on how to evaluate a RAG system covers how to measure retrieval once you can see it.

Tool calls and agent runs

Record the tool name, validated arguments, result status and duration, along with the step count and stop reason for the run. Log approvals and rejections too. This overlaps with the audit trail you need for human-in-the-loop approval, though audit records usually need stricter retention and tamper protection than debugging traces do.

Using OpenTelemetry's GenAI conventions

You do not have to invent your own attribute names. OpenTelemetry has semantic conventions for generative AI, which let different tools understand the same telemetry. They are maintained in a dedicated semantic-conventions-genai repository, and at the time of writing the client inference span and its attributes are marked as in development. Names may therefore change, so pin your versions and check the current specification.

The conventions name the span {gen_ai.operation.name} {gen_ai.request.model} and define attributes that include gen_ai.provider.name, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.response.finish_reasons. The illustrative wrapper below records them around a placeholder call_model function, whose reply object is assumed to carry these fields. Auto-instrumentation libraries may do this for you, in which case prefer them to hand-written code.

# Illustrative only. Assumes the OpenTelemetry API and a configured exporter.
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("my_llm_app")


def traced_chat(provider, model, messages, prompt_version, call_model):
    with tracer.start_as_current_span(f"chat {model}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", provider)
        span.set_attribute("gen_ai.request.model", model)
        # An application-specific attribute, not part of the conventions.
        span.set_attribute("app.prompt_version", prompt_version)
        try:
            reply = call_model(model, messages)
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            raise
        span.set_attribute("gen_ai.response.model", reply.model)
        span.set_attribute("gen_ai.usage.input_tokens", reply.input_tokens)
        span.set_attribute("gen_ai.usage.output_tokens", reply.output_tokens)
        span.set_attribute("gen_ai.response.finish_reasons", [reply.finish_reason])
        return reply

Note that message content is deliberately left out of this sample, which brings us to the hardest decision.

Prompts and privacy

Full prompts and responses are the most useful debugging data you can have, and often the most sensitive. They can contain personal details, confidential documents or credentials that a user pasted in. The OpenTelemetry conventions reflect this by treating content capture as opt-in: the configuration defaults to NO_CONTENT, with other settings controlled by OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT.

Make this a deliberate decision, with these options in mind:

Metadata only by default. Tokens, timings, IDs and statuses reveal a lot with little exposure.
Redaction. Remove known patterns before storage, accepting that automated redaction misses things.
Sampled or on-demand capture. Store content for a small share of traffic, or for flagged sessions.
Separate storage. Keep content apart from general telemetry, with stricter access and a shorter retention period.
Third-party review. A hosted tool receiving prompts is another data processor. Check your agreements and the regulations that apply to you.

Tell users what you retain, and make sure your deletion process reaches traces as well as your main database.

Metrics worth putting on a dashboard

Group your metrics by what they tell you, and split them by model, prompt version and feature so that a change can be pinned to its cause.

Area	Metrics	What a change may indicate
Reliability	Error and rate-limit counts, retries, timeouts	Provider problems or quota limits
Speed	Latency percentiles, time to first token	Longer context, slower models or queueing
Spend	Tokens per request, per user and per feature	Prompt growth, loops or misuse
Output health	Finish reasons, refusals, validation failures	Token limits set too low, or a prompt or model change
Agent behaviour	Steps per run, tool error rates, step-limit exits	Confusing tools or looping
Retrieval	Empty results, score distributions	Index problems or new kinds of question
User signals	Ratings, regenerations, handovers	Possible quality problems

Percentiles matter more than averages here, because a few long requests can hide behind a healthy mean. The guide to controlling LLM cost and latency covers what to do once you can see where the time and money go.

Measuring quality on live traffic

Traces show what happened but not whether it was good. Combine several weaker signals, as no single one is dependable:

Automated checks. Fast rules such as schema validity, citation presence or banned content can run on all traffic.
Model-based scoring. A second model can grade a sample of responses for attributes such as faithfulness to sources. It is imperfect and adds cost, so check its judgements against human labels.
User feedback. Valuable but sparse and skewed, since most people never click anything.
Human review. Reading a regular sample of real traces finds problems no metric was designed to catch.

Close the loop by adding interesting production failures, with appropriate consent and redaction, to your offline test set so that each one becomes a regression test.

Alerts and a debugging routine

Alert on things a person can act on: error rates, spend beyond a budget, latency against your target and sudden shifts in finish reasons or validation failures. Quality scores are noisier, so they are better reviewed as trends than used to wake anybody up. Thresholds should come from your own baseline, which you will only have after watching normal traffic for a while.

When a bad response is reported, a consistent routine helps. Find the trace, check what was retrieved, read the assembled prompt, compare the prompt and model version with the last known good run, then replay the input. Mark deployments on your dashboards too, as "what changed?" is usually the first useful question.

Choosing tooling

There are three broad routes: extend your existing OpenTelemetry-compatible observability stack, adopt an LLM-specific platform such as Langfuse or LangSmith, or begin with structured logs. LLM platforms tend to offer prompt views and evaluation features, while existing stacks keep everything in one place. Check current features yourself, and compare data residency, self-hosting options, sampling control and how easily you can export your data. Using open conventions where possible keeps a later move realistic.

A sensible first step is modest: add a trace ID, log model calls with tokens, timing and finish reason, and version your prompts. If you are serving an LLM app with FastAPI, this can sit in one shared client wrapper.

If you want this built into an LLM product, the generative AI and LLM development page describes what I can help with.

Frequently asked questions

What is the difference between LLM monitoring and observability?

Monitoring watches known measures, such as error rate and latency, and tells you that something has changed. Observability records enough detail, particularly linked traces, for you to investigate why a specific response went wrong, including problems you did not anticipate.

Should I log full prompts and responses in production?

Only as a deliberate decision. They are very useful for debugging but may contain personal or confidential data. Many teams can start with metadata only, then add redaction, sampling, restricted access and short retention for any stored content.

Do I need a dedicated LLM observability tool?

Not at the start. Structured logs or an existing OpenTelemetry setup can cover tokens, latency and traces. Dedicated tools become more attractive when you want prompt views and evaluation workflows, so judge them against your actual needs and data rules.

How is observability different from evaluation?

Evaluation usually tests a fixed set of examples before release, to compare versions. Observability looks at real traffic after release. They work best together, with production failures feeding the test set and test measures informing what you watch.

LLM observability and tracing: monitoring LLM apps in production