Controlling LLM Cost and Latency

Start by measuring

Optimising without data usually means working on the wrong thing. A slow answer may be caused by retrieval, a reranker, a tool call or a queue, not by the model. Before changing anything, record these for each model call and each step of a request:

Input tokens, output tokens and any cached tokens.
The model used, the feature it served and the outcome, including retries and failures.
Time to first token, total duration and time spent in each non-model step.
Your estimated cost per request and per completed user task.

Look at cost per completed task as well as per call, since a cheaper model that needs several retries may cost more overall. For latency, review high percentiles and not only the average, as the slowest requests shape how reliable a product feels. Group data by feature too: often one feature or a small group of unusually long conversations accounts for much of the total.

You also need an evaluation set or an equivalent test for your task. Every technique below can reduce quality, and without a test you will find out from your users.

Know what drives each one

Cost and latency are related, but they do not always move together. Hosted models are commonly priced by tokens, with output often priced differently from input, so check your provider's current pricing. Latency is shaped by the model, input size and especially how much output is generated, along with queueing and everything your application does around the call.

Technique	Helps cost	Helps latency	Main risk
Trimming context	Yes	Often	Removing something the model needed
Routing to a smaller model	Usually	Usually	Lower quality on hard cases
Response caching	Yes	Yes, on a hit	Stale or wrongly shared answers
Provider prompt caching	Usually	Often	Little benefit if prompts keep changing
Capping output	Yes	Yes	Answers cut off mid-way
Streaming	No	Perceived only	Harder validation and error handling
Batch processing	Often	No, it is slower	Unsuitable for interactive use

Send less to the model

Input is often the easiest place to find savings, as prompts tend to grow a little at a time.

Audit the system prompt. Remove old instructions, repeated rules and examples that no longer change results. Test each removal.
Manage conversation history. Sending every earlier turn makes long conversations steadily more expensive. Keep recent turns and summarise or drop older ones. Agent memory explains how to decide what to keep.
Retrieve less, but better. In RAG, more passages mean more tokens and sometimes more distraction. Better retrieval and reranking can let you send fewer passages, though a reranker adds its own latency.
Keep tool definitions and results small. Each tool definition uses input tokens, and large tool results stay in context for later turns. Load only the relevant tools, and return only the fields needed.
Strip noise. Remove markup, boilerplate and repeated headers from documents before they reach the prompt.

Use the right model for each task

Not all steps need your most capable model. Classifying a request, extracting simple fields and rewriting a query are often handled well by something smaller and faster, while complex reasoning may justify a larger one.

Static routing assigns a fixed model to each step or feature. It is simple and predictable, and it is a good place to begin. Dynamic routing chooses per request using rules or a small classifier, and adds its own errors and delay. Cascades try a cheap option first and escalate when a check fails or confidence is low, which works only if you have a reliable way to detect failure.

Also ask whether the step needs an LLM at all. Rules, a database lookup or a traditional classifier can be much faster for tasks with a clear structure.

Here is an illustrative cascade. The helper functions are placeholders:

# Illustrative only. Helpers and model labels are placeholders.
def answer_with_cascade(request):
    draft = call_model(model="small", prompt=request.prompt)
    if passes_checks(draft, request):
        return draft
    return call_model(model="large", prompt=request.prompt)

Cache carefully

Caching can save both money and time, but there are several kinds and they carry different risks.

Exact response caching. Store the result under a key made from the model, the full prompt and the relevant settings. It is safe and simple, and it works best when inputs repeat, such as when processing the same document again.
Semantic caching. Reuse an answer for a question with a similar meaning. This can raise the hit rate, but two similar questions can need different answers, so test thresholds on realistic data and consider it carefully for high-stakes uses.
Provider prompt caching. Many providers offer a way to reuse the processing of a repeated prompt prefix. Availability and rules vary, so check current documentation. To get the most from it, put stable content such as instructions and tool definitions first, and variable content last.
Embedding and retrieval caching. Avoid re-embedding unchanged documents, and cache repeated query embeddings where appropriate.

Two safety rules apply to all of these. Scope keys by whatever affects the answer, including the user or tenant and their permissions, so one person's private answer never reaches another. And give entries a way to expire or be invalidated when source content changes.

Control output

Generating output is often the slowest part of a call, so shorter answers help with both measures.

Ask for what you need. Specify length and format. "Return the label only" is cheaper than an explanation that nobody reads.
Set a maximum output limit. Treat it as a safety cap and handle a cut-off response explicitly, because a truncated answer or broken JSON should not pass as complete.
Use structured output where it fits. Fixed fields can reduce chatter and make parsing failures, and the retries they cause, less common.
Check reasoning settings. Some models offer settings that spend extra tokens and time on reasoning. Use the lowest level that passes your tests.

Reduce waiting at the application level

Stream long answers. Streaming does not shorten total generation time or reduce tokens, but people see progress sooner.
Run independent steps at the same time. Retrieval from two sources does not need to happen one after the other. Keep an eye on rate limits when you do this.
Move work out of the request. Summaries, tagging and evaluation can run in the background. Where a provider offers a batch option, it can suit work that does not need an immediate result.
Prepare ahead. Generate results that are likely to be needed before anyone asks, when the content is known and changes rarely.
Use timeouts and sensible retries. Retries with backoff and a limit prevent a brief fault from turning into a flood of requests. A fallback model can help when a provider is slow, as long as it has been tested too.
Keep connections alive and work close together. Reusing clients avoids repeated connection setup, and placing services near each other can reduce network time. Serving an LLM app with FastAPI shows how to share clients and avoid blocking in a Python service.

Keep agents within limits

Agents multiply everything above, because a single request can trigger many model calls, each carrying a growing context. A loop that goes wrong can run up costs quickly.

Set hard limits on steps, tool calls, tokens and time per run, enforced in code, and return a useful message when one is reached.
Detect repetition. Stop or escalate when the same call keeps returning the same result.
Be careful with extra agents. Every handoff adds calls and context. This is one of the costs to weigh when choosing between single and multi-agent designs.
Compact context during long runs. Summarise or remove old tool results that are no longer needed.

Add budgets and guardrails

Finally, protect the system from unexpected use. Set maximum input sizes, rate limits and per-user or per-tenant quotas. Set budget alerts, and decide in advance what happens when a limit is reached: refuse, queue or switch to a smaller model. Make the behaviour clear to users. Then review your data regularly, since new features and changing usage can undo earlier savings.

For help designing LLM features with these controls in place, see generative AI and LLM development.

Frequently asked questions

What is the first step in reducing LLM cost?

Measure tokens, model and cost by feature and by step, so you can see where the spending goes. Trimming unnecessary context and history is then often a good early change, tested against your evaluation set.

Does streaming reduce LLM latency?

It reduces the time before people see something, not the total generation time, and it does not reduce tokens. For long answers, it makes waiting feel much better.

Is semantic caching safe?

It can be, with care. Similar questions can need different answers, so test thresholds on realistic data, scope entries by user and permissions, and avoid it where a wrong reused answer would be costly.

How do I stop an agent from running up costs?

Enforce limits on steps, tool calls, tokens and time per run in code, detect repeated calls and add per-user quotas and budget alerts. Prompt instructions alone are not a reliable limit.

Controlling LLM cost and latency