Agent Memory: Short-Term and Long-Term State

Why agents need memory at all

An LLM call is stateless. The model only knows what is included in the current request, so anything an agent appears to remember is something your application stored and chose to send again. Agent memory is therefore a software design problem: what to keep, where to put it, when to bring it back and when to get rid of it.

The word "memory" gets applied to several different things, and mixing them up causes most of the confusion. The most useful first split is by lifetime:

Short-term state belongs to one run or conversation. It holds the current goal, recent messages, tool results and progress so far.
Long-term memory outlives the conversation. It holds facts, preferences or lessons that may be useful in a later session.

Many agents need only the first. Add long-term memory when a real use case calls for it, because it brings considerably more risk and maintenance.

Short-term state: the working memory of a run

Short-term state has two jobs that are easy to blur. It supplies context to the model, and it records where the workflow has got to. Separating these helps, as they have different needs.

The message history is the conversational record: user messages, model replies, tool calls and tool results. Structured state holds named fields that your code can read directly, such as the selected order ID, a list of completed steps or an approval status. Routing decisions and limits should depend on structured fields, not on a model's reading of a long transcript. The guide to building an orchestrator with LangGraph shows this kind of state in a graph-based workflow.

Keeping context under control

Context windows are finite, and longer contexts cost more and can distract the model. As a run grows, you have a few options:

Trim old messages. Simple and predictable, but it can lose an instruction given early on. Protect the system prompt, the goal and any open commitments.
Summarise older turns. This preserves more, but a summary can drop or distort a detail, and errors then persist. Keep the originals in storage in case you need them.
Compact tool results. A large tool response is often only needed briefly. Replace it afterwards with a short note and a reference for fetching the full version.
Move facts into structured state. A confirmed delivery address belongs in a field rather than somewhere in a transcript that may be trimmed.

When you trim, keep tool calls paired with their results, since many model APIs reject or misread a history in which one is missing.

Checkpoints are not the same as memory

A checkpoint saves a run's state so that it can resume after a crash or a pause for human approval. It is about durability: the same run continues from where it stopped. Long-term memory is about what a future, different run should know. The two can use the same database, but they should not be treated as one concept. Loading every previous checkpoint into a new conversation is not a memory strategy.

Long-term memory: what it can hold

Terms borrowed from psychology offer a handy way of sorting long-term memory, although the boundaries are loose and different tools use the words differently:

Type	What it stores	Example	Typical storage
Semantic	Facts and preferences	The user prefers metric units	Database rows, or a key-value or document store
Episodic	Records of past events	A summary of an earlier session and its outcome	Searchable log, often with embeddings
Procedural	How to do things	A refined instruction for a recurring task	Versioned prompts or configuration

Do not confuse any of these with a knowledge base. A shared body of documents that every user queries is the territory of RAG. Retrieval techniques may overlap, but memory is usually written by the agent during use and is often specific to a person, which brings its own problems around scope, correctness and deletion.

Deciding what to write

Writing is where long-term memory most often goes wrong. Save everything and retrieval fills up with noise. Save carelessly and a misunderstanding becomes a lasting belief. Some practical policies:

Prefer explicit signals. "Always send me summaries in French" is a much better candidate than something the model inferred from a single message.
Treat inferred memories with suspicion. Mark their source and confidence, or ask the user to confirm before they are saved.
Choose when to write. Writing during the conversation makes memories available immediately but slows the main path. Writing in a background job allows better consolidation but introduces a delay. Either can be reasonable.
Exclude sensitive data by default. Decide which categories must never be stored, and enforce this in code, not only in a prompt.
Record provenance. Store who or what the memory came from, when, and which conversation produced it.

# Illustrative only. A memory record with scope and provenance.
from dataclasses import dataclass
from datetime import datetime
from typing import Optional

@dataclass
class MemoryRecord:
    id: str
    tenant_id: str
    user_id: str
    kind: str             # "preference", "fact" or "episode"
    content: str
    source: str           # "user_stated" or "inferred"
    source_run_id: str
    created_at: datetime
    expires_at: Optional[datetime] = None
    superseded_by: Optional[str] = None  # id of the replacement

Retrieving the right memories

Some memories should always be present, such as a small profile of confirmed preferences. Others should be fetched only when relevant. Semantic search by embedding is a common choice for that second group, and it can usefully be combined with filters on type or date, and with a weighting that favours recent items. Exact lookup by key is simpler and more reliable where the application knows what it needs.

Apply access filters in the query itself, before any similarity ranking, so that one user's memories can never appear for another. Keep the amount injected small, and label it as background information from earlier sessions, so that the model does not treat it as the user's current request.

Updates, conflicts and forgetting

People move, change jobs and change their minds. If a store only ever appends, an agent will eventually hold two contradictory facts with no way to choose between them. Before saving a new memory, look for existing related ones and decide whether to add, update or replace. Keeping a superseded record with a link to its replacement leaves an audit trail without letting the old version be retrieved.

Forgetting should be a planned feature:

Expiry. Some information is only useful for a limited time, so it should carry an expiry date.
User control. Let people see, correct and delete what is held about them, where appropriate for the product.
Real deletion. Check that deleted content also leaves any embedding indexes, summaries, caches and backups, in line with your retention obligations.

Security risks specific to memory

Memory turns a one-off prompt injection into a persistent one. If an agent reads a malicious web page and saves an instruction from it, that instruction could influence every later session. This is often called memory poisoning, and the defences are similar to those in the guide on LLM guardrails and prompt injection defence: do not write to memory directly from untrusted content without review or restriction, record provenance, and limit what any retrieved memory can trigger. A memory should never be able to grant a permission.

Memory in multi-agent systems

When several agents are involved, decide what is shared and what stays private. Passing every agent the entire history is simple, but it wastes context and leaks details an agent has no need to see. A shared structured state, with defined owners for each field, generally behaves better than a shared transcript. Concurrent writes need a clear rule as well, such as a single writer per field or a defined merge function, or updates can be silently lost.

Testing memory behaviour

Recall across sessions. Tell the agent something, start a new session and check that it uses the fact when relevant.
Restraint. Check that it does not bring up remembered facts when they are irrelevant.
Updates. Change a preference and confirm that the newer value wins.
Isolation. Confirm that one user can never retrieve another's memories.
Deletion. Delete a memory and confirm that it no longer affects behaviour.
Long runs. Push past the point at which trimming or summarising begins, and check that the goal and constraints survive.

If you are designing state, checkpointing and routing for an agent workflow, see AI orchestration with LangGraph.

Context also has a direct effect on spending and response times, a topic covered in controlling LLM cost and latency.

Frequently asked questions

What is the difference between short-term and long-term agent memory?

Short-term state belongs to one run or conversation and holds things such as recent messages and progress. Long-term memory lasts beyond it, holding facts, preferences or records that may be useful in later sessions.

Does every AI agent need long-term memory?

No. Many agents work well with short-term state alone. Long-term memory adds work around correctness, privacy and deletion, so add it when a specific use case benefits from information carrying over between sessions.

Is agent memory the same as RAG?

Not quite. They can use similar retrieval methods, but RAG usually searches a maintained body of documents, while agent memory is typically written during use and is often specific to a user or task.

What is memory poisoning?

It is when malicious or false content gets saved to an agent's long-term memory and then influences later sessions. Restricting writes from untrusted content, recording provenance and limiting what a memory can trigger all reduce the risk.

Agent memory: short-term and long-term state