LLM caching strategies: exact and semantic caching and their risks

What this guide covers

Caching is one of several levers covered in controlling LLM cost and latency, which explains where it fits among the others. This guide takes a closer look at the part that tends to go wrong: deciding when a stored answer is really the right answer for a new request.

A cache makes a claim each time it returns a hit, namely that this earlier output is correct for this request, for this person, right now. Everything below is a way of making that claim true.

Three things called caching

The word covers different mechanisms, and they carry very different risks:

• Exact response caching returns a stored output when a key built from the request matches exactly. The model is not called.
• Semantic caching returns a stored output when a new request is close enough in embedding space to an earlier one. Again the model is not called, but now the match is a judgement call.
• Provider prompt caching reuses the processing of a repeated prompt prefix, and it does not return old answers. OpenAI's prompt caching guide says the model still has to process new input and generate a new response, and that the entire rendered prefix must match. Other providers have their own rules, so read their documentation.

Because prompt caching still produces a fresh answer, it does not have the wrong-answer risks discussed here. The rest of this guide is about the first two.

Exact caching: the key is the design

An exact cache can only be wrong in one way: when two requests share a key but should have different answers. So the key must include everything that can change the output or who may see it:

• The model identifier, including a pinned version if the provider offers one.
• The complete rendered messages, including the system prompt and any retrieved passages.
• Parameters such as temperature, maximum tokens, tool definitions and response schema.
• The tenant, and the user or permission scope where answers differ by access.
• A version number of your own, which you can raise to discard everything at once.

# Illustrative only. Field choices depend on your application.
import hashlib
import json

CACHE_VERSION = "v3"

def cache_key(model, messages, params, tenant_id, permission_scope):
    payload = {
        "version": CACHE_VERSION,
        "model": model,
        "messages": messages,
        "params": params,
        "tenant": tenant_id,
        "scope": permission_scope,
    }
    canonical = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

This sample is illustrative. Serialising with sorted keys matters, as otherwise two equivalent requests can hash differently. Watch out too for anything that varies needlessly: a timestamp in the system prompt or a request ID in the messages will mean that the cache never hits, and the failure is silent.

Exact caching suits repeatable work, such as re-running a pipeline over unchanged documents, classifying duplicate records or replaying test runs. Conversational traffic rarely repeats exactly, particularly when a long history is part of the prompt. Normalising input by trimming whitespace is safe. Lowercasing or removing punctuation changes the text, so treat it as a decision to test, since these changes can alter meaning in codes, names and identifiers.

Do not cache everything that succeeds

Decide what is eligible. Refusals, truncated output, failed validation and error messages should not be stored, or one bad moment will be served repeatedly. Tool calls that change something must not be replayed from a cache at all, since the second request would skip the action while reporting that it happened. Temperature matters too: if you want varied output for creative work, a cache removes that variety.

Semantic caching: a classifier in disguise

A semantic cache embeds the incoming request, searches earlier requests and returns the stored answer if the nearest match is within a distance threshold. This is effectively a binary classifier answering "do these two requests have the same answer?", and it should be evaluated as one.

Embeddings measure similarity of meaning, and that is not the same as interchangeability. These pairs may sit close together yet need different answers:

• "How do I cancel my annual plan?" and "How do I cancel my monthly plan?"
• "Can I take this medication with alcohol?" and "Can I take this medication without food?"
• "Is the API rate limit higher on Pro?" and "Is the API rate limit lower on Pro?"
• "What's our refund policy?" asked by a customer in two countries with different policies.

Short follow-ups are especially dangerous. "What about the second one?" means something different in every conversation, so either rewrite it into a standalone question first or do not use the cache for it. Likewise, a request whose answer depends on retrieved documents or personal context is only comparable if that context is comparable too.

Choosing a threshold

No threshold is correct in general, as it depends on your embedding model and on your traffic. Do not copy a value from a tutorial. Instead:

1. Sample real request pairs across a range of distances, including near misses like those above.
2. Have someone who knows the domain label each pair: could the stored answer be served for the new question?
3. Measure false hits and useful hits at several thresholds, then choose based on the cost of a wrong answer.
4. Repeat when you change embedding model, as distances from different models are not comparable. Treat that change as a new cache.

If no threshold gives acceptably few false hits with a useful number of hits, the honest conclusion is that semantic caching does not fit this feature. A review step can help during tuning: for a sample of hits, call the model anyway and compare the results.

Safer designs

• Curated answers. Only approved answers to known questions are served, so a match can never return a poor generated answer. This resembles an FAQ with fuzzy matching.
• Match, then verify. Use similarity to find a candidate, then check it with a small classifier or a model call. This costs more per request, and the saving must still justify it.
• Cache retrieval instead of answers. Reusing search results for similar queries limits what a false hit can do, because the model still composes an answer to the actual question.
• Restrict by intent. Allow the cache only for categories where it has been tested, such as general product questions, and not for account-specific ones.

The risks in production

Leaks between users

A shared cache is a route around your permission system. An answer built from documents that one user could read should not be returned to someone who could not read them, even when the question is the same. Partition by tenant and permission scope, and filter on that scope in the lookup. For semantic caches, this must be a hard filter rather than a similarity signal. RedisVL's semantic cache guide shows one approach, using filterable fields so that a lookup only considers entries for a given user. The wider design is in RAG access control and permissions. Where an answer contains personal data, also treat cached copies as stored personal data, with suitable retention and deletion.

Stale answers

A time limit is the usual tool, but it alone leaves answers wrong until they expire. Add invalidation by storing the source document IDs and versions with each entry, then removing entries when a source changes. For a RAG system, including a content version in the key achieves something similar.

Read how your library treats expiry. The same RedisVL guide notes that TTL defaults to none, so entries persist, and that its check method refreshes the TTL on matched entries. With a sliding window such as this, a frequently requested answer may never expire, which is exactly the entry you most want to keep current. Consider an absolute maximum age. Also bump the cache version when you change a prompt or model, otherwise old behaviour lingers after the release.

Poisoning

If one user's interaction can fill an entry that others receive, a successful prompt injection can spread. An attacker who manipulates a model into a bad answer to a common question may have that answer served to everyone who asks something similar. This is a direct consequence of sharing, and it is my reasoning rather than a statistic about attacks, but it should affect your design. Validate outputs before storing them, prevent shared entries being created from requests with personal or untrusted context, and provide a way to remove entries quickly. LLM guardrails and prompt injection defence covers the underlying problem.

Hidden quality problems

Caching can hide things. A wrong answer repeated from a cache produces no new errors, and a model upgrade may look like it changed nothing because most traffic never reached it. So record each hit, the matched entry and its distance in your traces, and give users' negative feedback a way to find and evict the entry involved. Bypass the cache during evaluation runs, unless the cache is what you are testing.

Choosing an approach

1. Start with provider prompt caching if prompts have a long stable prefix, as it has no wrong-answer risk.
2. Add exact caching where your logs show that requests really do repeat.
3. Look at your logs to see how many requests are near-duplicates with the same answer. If there are few, stop there.
4. Only then trial semantic caching on a narrow, low-risk category, in shadow mode first, comparing what it would have returned with what the model produced.

For help designing LLM features with these safeguards, see generative AI and LLM development.

Frequently asked questions

What is the difference between exact and semantic caching?

Exact caching returns a stored answer only when a key built from the full request matches. Semantic caching returns one when a new request is close in embedding space to an earlier one, which raises hit rates but can return the wrong answer.

Is provider prompt caching the same as response caching?

No. Prompt caching reuses the processing of a repeated prompt prefix, and the model still generates a new response. Response caching skips the model and returns a stored output, so it can serve stale or wrongly matched answers.

What similarity threshold should a semantic cache use?

There is no universal value, because it depends on the embedding model and your traffic. Label real request pairs, including near misses, measure false hits at several thresholds and choose according to what a wrong answer would cost.

Should LLM responses be cached per user?

Scope them by whatever changes the answer or who may see it. Personal or permission-dependent answers need user or permission scope, while general public answers can be shared within a tenant or more widely if that is safe.