Streaming LLM responses to a web UI

What streaming does and does not improve

A language model generates text gradually, and a complete response can take a while. If your interface waits for the end, users look at a spinner. If it streams, they see the first words soon after generation starts and can begin reading or decide to stop.

Streaming improves perceived responsiveness, not the total time taken. The delay before the first token may also still be noticeable, particularly if your backend first performs retrieval or tool calls. Showing a status such as "searching documents" during that gap is part of the same design. This guide goes deeper on the mechanics mentioned in the broader guide to adding an LLM feature to a full stack app.

The path from model to screen

A typical streamed response passes through several stages:

1. The browser sends the prompt to your backend, never directly to the model provider, so that API keys stay private and your checks cannot be bypassed.
2. The backend authenticates the user, applies limits, assembles the context and opens a streaming call to the model.
3. As chunks of text arrive, the backend forwards them in a format of your own.
4. The browser reads those events, updates its state and renders the text.
5. On completion, the backend stores the final message and usage data, then signals the end of the stream.

Use your own event format at step 3. It keeps your front-end independent of any one model provider, and it gives you a place for status updates, sources and errors.

Choosing a transport

A common choice for chat is to send a POST request using fetch and have the server respond in the server-sent events format, which allows a request body and an authorisation header. Long prompts should not go in a URL in any case, since URLs tend to end up in logs. If a separate step creates the run, EventSource can connect to a URL for that run, using cookie-based authentication.

Designing the events

Forwarding raw text works until you need anything else. Typed events keep the protocol clear:

event: status
data: {"message": "Searching documents"}

event: token
data: {"text": "The refund "}

event: sources
data: {"items": [{"id": "doc-12", "title": "Refund policy"}]}

event: done
data: {"message_id": "m_123", "finish_reason": "stop"}

The example is illustrative, and the field names are yours to define. Encoding each payload as JSON matters because a raw newline in generated text would otherwise break the line-based format. A blank line ends each event. An explicit done event also lets the client distinguish a complete response from a connection that simply dropped.

The backend

The server side is essentially a generator that forwards chunks, with care taken at the edges. Here is a framework-neutral sketch using placeholder functions:

# Illustrative only. model_stream, save_message and log_failure
# are placeholders; model_stream is an async generator.
import json

def sse(event, payload):
    return f"event: {event}\ndata: {json.dumps(payload)}\n\n"

async def stream_answer(request, prompt):
    parts = []
    finished = False
    stream = model_stream(prompt)
    try:
        async for text in stream:
            if await request.is_disconnected():
                return
            parts.append(text)
            yield sse("token", {"text": text})
        finished = True
    except Exception:
        log_failure(request)
        yield sse("error", {"message": "The response could not be completed."})
    finally:
        await stream.aclose()  # leaving the loop does not close it
        message_id = await save_message("".join(parts), complete=finished)

    if finished:
        yield sse("done", {"message_id": message_id, "finish_reason": "stop"})

Details to handle:

• Headers commit early. Once streaming has begun, the HTTP status has already been sent. Later failures have to be reported as events, so complete authentication and validation first.
• Disconnects are only noticed between chunks. The check above runs when a chunk arrives, so a stalled model call also needs a timeout, and some servers offer a cancellation signal that fires sooner.
• Do not expose raw errors. Log the detail on the server and send a safe message to the browser.
• Use non-blocking calls. In an async server, a blocking call inside the stream can hold up other users' requests.
• Save partial output deliberately. Decide whether stopped responses are stored and marked as incomplete.
• Enforce limits. Set a maximum output length, a generation timeout and a per-user limit on concurrent streams, because each one holds a connection open.

For how a streaming route fits into a Python service as a whole, see serving an LLM app with FastAPI.

Reading the stream in the browser

Network chunks do not line up with events. One read can contain half an event or several at once, and a multi-byte character can be split across reads. The parser therefore needs a buffer, and the decoder needs to be told more data is coming:

// Illustrative only. parseEvent and handleEvent are your own functions.
async function readStream(prompt, signal) {
  const response = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt }),
    signal,
  });
  if (!response.ok) throw new Error("Request failed");

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    const events = buffer.split("\n\n");
    buffer = events.pop(); // keep any incomplete event
    events.forEach((raw) => handleEvent(parseEvent(raw)));
  }
}

If the loop ends without a done event, treat the response as interrupted, even though no exception occurred.

Rendering partial output

Partial text brings its own interface problems:

• Batch updates. Re-rendering the whole message for each small chunk can make long responses sluggish. Collect incoming text and update the display on an animation frame or a short timer.
• Incomplete Markdown. An unclosed code fence or half a link will look broken until the rest arrives. Use a parser that tolerates this, and accept that the formatting may shift as text arrives.
• Sanitise before rendering. Model output is untrusted. Sanitise any HTML generated from Markdown, on each update, and consider restricting links and images. LLM guardrails and prompt injection defence explains how a rendered image can leak data.
• Respect scrolling. Follow new text only while the user is at the bottom. If they scroll up to read, stop moving the page.
• Consider accessibility. Announcing each fragment through a live region can overwhelm screen reader users. Announcing status changes and the finished message is usually more workable. Test this with real assistive technology.
• Prevent overflow. Long code lines should scroll inside their own block, so the page does not widen on a phone.

Cancellation and interrupted streams

A stop button should do more than hide the text. Calling abort() on an AbortController ends the browser's request, and the server should notice the disconnect and cancel the upstream call. Otherwise the model may keep generating text that nobody will read.

Disconnects happen unintentionally as well, on phones especially. For short answers, showing the partial text with a retry button may be enough. For long or expensive runs, it may be better to separate the work from the connection: a background job writes numbered events to storage, and the client reads from its last seen position after reconnecting. Server-sent events support this through event IDs and the Last-Event-ID header sent by EventSource on reconnection. Agent runs that pause for review, as discussed in human-in-the-loop approval for AI agents, generally need this separation.

Proxies and deployment

The most common surprise is a stream that works locally and arrives all at once in production. Check each hop:

• Reverse proxies and CDNs may buffer responses unless configured otherwise.
• Compression can hold back small chunks while it collects enough data to compress.
• Idle timeouts may close a connection during a long silence, such as a slow tool call. Sending a periodic comment line as a heartbeat can keep it open.
• Serverless limits on duration or response streaming vary by platform, so check them against your longest expected run.

Testing a stream

Replace the model with a stub that emits a scripted sequence, including splits inside events and multi-byte characters. Then test a mid-stream error, a dropped connection with no final event, a stop click, a slow first token and very long output. Network throttling in browser developer tools is useful for this, and a final check against the deployed environment is essential.

For help building this layer, see full stack development for AI products.

Frequently asked questions

Should I use SSE or WebSockets to stream LLM responses?

Server-sent events are often enough, because the output flows one way and they run over ordinary HTTP. WebSockets make more sense if you already use them or need frequent two-way messages during a response, such as collaborative or voice features.

Why does my LLM stream arrive all at once in production?

Something between the server and browser is probably buffering the response, such as a reverse proxy, load balancer, CDN or compression layer. Test the deployed environment and check each hop's settings for streaming responses.

Can I stream structured JSON from an LLM?

You can, but partial JSON is not valid until it is complete. A tolerant parser can show progress, but any logic that depends on the data should wait for the finished output and validate it against a schema.

How do I stop generation when the user clicks stop?

Abort the browser request, detect the disconnect on the server and cancel the upstream model call. Check your provider's behaviour too, since closing the connection may not always stop generation or billing in the way you expect.