Adding an LLM Feature to a Full Stack App

Start with the feature, not the model

Before writing any code, describe the feature in plain terms. What goes in, what comes out, and what does the user do with the result? "Add AI" is not a specification, whereas "draft a reply to this ticket for the agent to edit" is. That sentence also reveals something important: a person reviews the result, which changes how much risk an error carries.

Also ask whether an LLM is necessary at all. If a search filter, a template or a rule would solve the problem, it will be cheaper, faster and more predictable. An LLM earns its place when the input is varied language and fixed logic falls short.

A typical architecture

A common structure places an AI service layer between your application and the model provider:

Browser UI
    |
Backend API  (auth, validation, rate limits)
    |
AI service layer  (prompts, context, output checks)
    |
Model provider or self-hosted model

The most important rule follows from this diagram: call the model from the backend, not the browser. Anything in the browser can be read by users, including API keys. The backend also gives you one enforceable place for authentication, quotas, logging and policy checks.

Keeping LLM logic in its own layer has further benefits. Prompts are kept out of your route handlers, a provider can be swapped with limited changes, and the logic can be tested independently. If your main application is not written in Python but your AI code is, that layer can run as a separate service.

Designing the backend

If the feature is an agent that runs for a long time or takes actions, also see building a backend API for an AI agent in Python.

Build the prompt on the server

Treat prompts as code: keep them in version control and test them. Keep your instructions clearly separated from user-provided content, and fetch data with the permissions of the current user, so that the model never receives content that user could not otherwise see.

Use structured output when code consumes the result

If your application needs fields rather than prose, ask for a defined format and validate what comes back. Many providers offer features for schema-constrained output, but your code should still validate, as the result may be semantically wrong even when the format is correct.

import json

CATEGORIES = {"billing", "technical", "other"}

def triage(ticket_text):
    raw = call_llm(build_prompt(ticket_text))
    try:
        data = json.loads(raw)
        if data["category"] not in CATEGORIES:
            return None
        if not isinstance(data["needs_human"], bool):
            return None  # bool("false") would be True
        return {
            "category": data["category"],
            "summary": str(data["summary"]),
            "needs_human": data["needs_human"],
        }
    except (json.JSONDecodeError, KeyError, TypeError):
        return None  # retry, or use a safe fallback

Here call_llm and build_prompt stand in for your own code, and a schema validation library can replace the manual checks. Decide what a failure means in your application: a limited retry, a default value or a visible error are all legitimate.

Plan for failure

Calls to a model can be slow, can be rate limited and can fail. Your backend should have:

Timeouts so that a stalled call cannot hold up a request indefinitely.
Limited retries with backoff for temporary errors, without repeating calls that fail validation endlessly.
A fallback so that the rest of the page still works when the AI feature does not.
Background jobs for work that will outlast a normal request, such as processing a large document.

Streaming responses

LLM responses can take a noticeable time, and waiting in front of a blank screen feels longer than watching text appear. Streaming sends partial output as it is generated. Server-sent events are a common choice, because the traffic is one-way from server to browser, while WebSockets may be preferable if you already use them or need two-way communication. The guide on streaming LLM responses to a web UI covers the details.

Streaming also complicates some things. Output can only be fully validated once it is complete, so it suits free text better than JSON that drives logic. An error can occur halfway through a response, so the UI needs a partial-failure state. Proxies may buffer responses, which should be checked in your deployed environment. Cancel the upstream call if the user navigates away, so that you do not pay for output nobody will read.

Designing the interface

Interface decisions affect how much people can trust an LLM feature, and the UI should make the model's limitations manageable.

Show every state. Idle, loading, streaming, complete, failed and cancelled should all have a defined appearance.
Make editing easy. A draft that can be changed is much more forgiving than an automatic action.
Allow stopping and retrying. Long responses should have a stop control.
Show sources where relevant. Links to source material let users verify answers. If grounding on your own content is required, read what RAG is and when to use it.
Be clear that the content is AI-generated, and that it may need checking.
Confirm consequential actions. Sending, deleting and paying should require a deliberate user action.
Render output safely. Treat model output as untrusted and sanitise it before displaying it as HTML or Markdown, to avoid script injection.

Security and privacy

LLM features introduce some new risks and sharpen some older ones.

Prompt injection. Users, or content the model reads, may include text designed to override your instructions. There is no complete defence at the prompt level, so limit the damage possible: enforce permissions in code, restrict what tools can do and require approval for sensitive actions. See LLM guardrails and prompt injection defence.
Data handling. Know what data you are sending to the model provider, review its terms for retention and training use, and minimise personal or confidential information.
Logging. Prompts and responses are useful for debugging, but they may contain sensitive data. Set access controls and retention limits.
Abuse. A public endpoint connected to a paid model is an attractive target. Require authentication where possible, and apply per-user rate limits and input size limits.

Managing cost and latency

Most hosted model pricing is based on usage, often on the tokens sent and received, so costs follow your prompt size and traffic. Useful controls include:

Recording token usage per feature and per user, so that you can see where spending goes.
Limiting output length and trimming unnecessary context.
Using a smaller model for simple tasks and reserving a larger one for those that need it.
Caching results for repeated identical requests, where that is safe, while making sure one user's private results are never served to another.
Setting spending alerts and limits before launch.

For a fuller treatment of these techniques, see controlling LLM cost and latency.

Testing an LLM feature

Because the output varies, ordinary exact-match assertions are not enough on their own. Use a combination of approaches:

Standard tests with a stubbed model. Cover routes, permissions, validation, fallbacks and UI states without any live model calls. These can run quickly in continuous integration.
Evaluation sets. Maintain realistic inputs, including edge cases and hostile inputs, and score the results against defined criteria each time a prompt or model changes.
Production feedback. Monitor errors, latency and cost, and provide a simple way for users to flag poor results. With appropriate privacy handling, flagged cases can become new test cases.

Rolling it out

Put the feature behind a flag and release it to a small group first, watching for failure cases your test set missed. Pin a specific model version if your provider allows it, as behaviour can change when a model is updated, and evaluate before you move to a newer one. Keep a way to disable the feature without a full deployment, so that a provider outage or quality problem has a simple response.

A short checklist

The task, its users and acceptable failure are defined.
Model calls and keys remain on the backend.
Authentication, permissions and rate limits are enforced in code.
Outputs are validated and rendered safely.
Timeouts, retries and a fallback exist.
The UI handles streaming, errors and cancellation.
An evaluation set runs when prompts or models change.
Usage and cost are monitored, with limits in place.

For help with either side of this work, see generative AI and LLM development and full stack development.

Frequently asked questions

Can I call an LLM API directly from the front-end?

It is generally unsafe for production, because API keys in browser code can be extracted. Route calls through your backend, where you can also apply authentication, rate limits and logging.

Should I stream LLM responses?

Streaming usually suits longer free-text responses, because users see progress straight away. For short responses, or structured output that must be validated before use, a normal request is often simpler.

What security risks does an LLM feature add?

The main ones are prompt injection, sensitive data sent to a model provider or kept in logs, unsafe rendering of model output, and abuse of a paid endpoint. Permissions in code, data minimisation, sanitising and per-user limits address them.

Do I need a framework to add an LLM feature?

Not for a simple feature, where a provider client and your own code may be enough. Frameworks become more useful for retrieval pipelines or complex agent workflows, so let the actual requirements decide.

Adding an LLM feature to a full stack app