LLM Guardrails and Prompt Injection Defence

Two problems that are often mixed up

"Guardrails" covers two related but different concerns, and it helps to separate them:

Quality and policy. Keeping the system on topic, avoiding harmful content and returning output in the correct format. Here the threat is mostly the model's own mistakes.
Security. Stopping someone from using the model to reach data or trigger actions they should not. Here there is an adversary, who will adapt to your defences.

A classifier that catches most off-topic messages may be perfectly adequate for the first concern. A control that an attacker can defeat by rewording their request is not adequate for the second. Much of the design work lies in knowing which situation you are in.

What prompt injection is

A language model receives its instructions and its data as one sequence of text. Delimiters and instructions such as "treat the following as data" can help, but they do not create a hard separation of the kind that parameterised queries provide against SQL injection. Text that looks like an instruction may be followed, wherever it came from.

There are two broad forms:

Direct injection. A user types something intended to override your instructions, such as asking the assistant to disregard its rules or to adopt a role without them.
Indirect injection. The instructions are hidden in content the model reads on someone's behalf: a web page, an email, a support ticket or a RAG document. The person using the system may be entirely innocent, and may never see the text.

Indirect injection is usually the greater concern in agentic systems. Consider an assistant that can read email and send messages. An attacker who emails its user a hidden instruction to forward private messages has, in effect, attempted to issue a command using that user's permissions.

The dangerous combination

Risk rises sharply when one workflow combines three capabilities:

It reads untrusted content.
It has access to private data.
It has a way to send information out or take action.

With all three present, a successful injection can lead to data theft or unwanted actions. Removing any one of them greatly narrows what an attacker can achieve. This is a useful test to apply to every workflow, and it is often more productive than trying to perfect a filter.

A layered design

No single control is sufficient, so the usual approach is to layer them, while being clear about which layers are hard boundaries and which only reduce likelihood.

Layer	Examples	Strength
Input checks	Length limits, injection classifiers, topic checks	Reduces likelihood; can be evaded
Prompt design	Clear roles, marked untrusted content	Reduces likelihood; can be evaded
Output checks	Schema validation, sanitising, sensitive data scans	Strong for format; partial for meaning
Permissions in code	Per-user access, narrow tools, argument validation	A hard boundary if implemented correctly
Human approval	Review before sensitive actions	Strong, but subject to human error
Monitoring	Logs, alerts, rate limits	Detects problems and limits their scale

Limit what the model can do

The most reliable defence assumes that the model will sometimes be manipulated, and makes sure that this does not matter very much. Much of this comes down to how an agent's tools are designed.

Use the user's permissions, not the system's. A tool that runs with a broad service account lets any user who manipulates the model reach everything that account can reach. Pass the authenticated user's identity from your application, never from model output.
Provide narrow tools. A get_order_status tool is much easier to protect than a general database query tool.
Validate arguments in code. Treat tool arguments like any other untrusted request: check their types, ranges, ownership and allowed destinations.
Filter retrieval by access rights before content reaches the model. A prompt instructing the model not to reveal certain documents is not an access control.
Separate risky work. A component that reads untrusted content can be given no sensitive tools, and it can return only constrained output, such as a category from a fixed list. See single agent vs multi-agent systems for this kind of split.
Require approval for consequential actions. The guide on human-in-the-loop approval covers this in detail.

The enforcement point looks like this, using placeholder functions:

# Illustrative only. The user comes from your auth layer, never the model.
def run_refund_tool(args, user):
    order = get_order(args["order_id"])

    if order is None or not user_can_access(user, order):
        return tool_error("Order not available.")

    if not valid_refund_amount(args["amount"], order):
        return tool_error("Amount not permitted.")

    return request_human_approval("refund", order.id, args["amount"])

Whatever the model was persuaded to request, this function cannot act on another user's order or exceed the permitted amount, and it can do no more than ask a person for approval.

Agents that query databases are a clear example of this principle, and text-to-SQL agents: risks and safeguards applies it to generated SQL.

Handle model output as untrusted

Output deserves the same suspicion as input, because it may carry an attacker's text straight into another system.

Web rendering. Sanitise HTML generated from model output, in order to prevent cross-site scripting.
Data leaks through URLs. If your interface automatically loads Markdown images, a manipulated model could produce an image URL containing private data in its query string, and the browser would send it to the attacker's server just by loading the image. Restrict images and links to approved domains. A content security policy provides a further layer.
Downstream execution. Do not pass model output directly to a shell, an interpreter or a database. If generated code or queries are part of the design, run them with minimal permissions in a sandbox or a read-only role.
Structured output. Validating against a schema catches malformed data. A well-formed value can still be malicious or wrong, so the content needs checking as well.

Where classifiers and filters fit

Input and output classifiers are still useful. They can stop casual attempts, keep a product on topic, flag activity for review and reduce the amount of abuse reaching your model. They do, however, have false positives, which block legitimate users, and false negatives, which let attacks through. Rephrasing, other languages and encodings can all be used to get around them.

Measure their error rates on your own traffic and test set, and consider the added latency. If a filter is the only thing between an attacker and a damaging action, redesign that part of the system so a filter miss is survivable. If you stream responses, also note that an output check on complete text cannot stop what has already been displayed; streaming LLM responses to a web UI covers that delivery path.

Prompt design still helps

A well-written prompt will not create a security boundary, but it can improve normal behaviour and raise the effort needed for simple attacks. State the model's task and limits clearly. Mark retrieved content, tell the model that it is reference material rather than instructions, and describe what to do when it cannot help. Keep credentials and sensitive rules out of the prompt, as a system prompt may be revealed.

Test like an attacker

Add adversarial cases to your regular evaluation so that defences are measured and regressions show up:

Direct attempts to override instructions, in several wordings and languages.
Instructions hidden in documents, emails and web pages your system might read.
Requests for another user's data, including through tools.
Attempts to make the model emit links or images containing private data.
Tool arguments outside the permitted range.

Judge the outcome by the effect, not only the text: did a forbidden action occur, or did private data reach an unauthorised place? The security checks can sit in the same test process described in how to evaluate a RAG system. A passing suite shows only that those attacks failed, so periodic manual testing by someone trying to break the system remains worthwhile.

Plan for the failure

Assume that something will eventually get through. Keep logs that let you reconstruct what the model saw, what it requested and what ran, while handling personal data appropriately. Set rate and spending limits to contain abuse. Make sure you can disable a tool or feature quickly and can revoke credentials. Where a matter has legal or regulatory implications, seek appropriate specialist advice, as this guide is general engineering information.

For help designing these controls into an application, see generative AI and LLM development.

Frequently asked questions

What is prompt injection?

Prompt injection is when text supplied to a language model causes it to follow instructions its developers did not intend. It may come directly from a user or indirectly from content the model reads, such as a document, email or web page.

Can prompt injection be fully prevented?

Not with current general techniques, as models process instructions and data as one stream of text. Filters and careful prompts reduce the likelihood, so systems should also limit what a successfully manipulated model is able to do.

What are LLM guardrails?

Guardrails are controls around a model that check inputs, constrain outputs and limit actions. They range from topic classifiers and schema validation to tool permissions and human approval. The controls enforced in ordinary code are the most dependable.

Is a system prompt secret?

It should not be treated as secret. A model may reveal or paraphrase it when prompted in the right way. Do not place credentials or sensitive logic there, and enforce access rules in the application instead.

LLM guardrails and prompt injection defence