Testing AI agents: how to handle non-deterministic behaviour

Why ordinary tests fall short

A conventional test expects one output for one input. An agent breaks that assumption in several ways. A model may phrase an answer differently on each run. It may choose another valid sequence of tool calls. Providers can update models, tools can return changing data, and a small prompt edit can alter behaviour far from where you made it. Asserting on exact text therefore produces tests that fail for harmless reasons and pass for meaningless ones.

The answer is not to give up on assertions. Much of an agent is ordinary software that should behave identically on every run: the loop, the tool code, the permission checks and the limits. Only part of the system is truly uncertain. Testing well begins with drawing that line. If you are new to the structure involved, how to build an AI agent in Python describes the loop these tests exercise.

The code samples below are illustrative, using pytest and placeholder names of my own for the agent's parts.

Think in layers

A workable test strategy has four layers, running from cheap and strict to expensive and statistical.

Layer 1: test the deterministic parts

Test your tools as ordinary functions, without any model. Give special attention to the safety rules, which must hold however the model behaves: a refund above the limit is rejected, and a user cannot read another customer's order. Also test how tools handle the malformed arguments a model might send. The guide on tool calling for AI agents covers how to design tools so that these errors are safe and useful.

# Illustrative only. issue_refund and PermissionDenied belong to your own code.
import pytest

@pytest.mark.parametrize("amount, allowed", [(50, True), (500, False), (-5, False)])
def test_refund_limits(amount, allowed):
    if allowed:
        assert issue_refund("order-1", amount, approved_limit=100).ok
    else:
        with pytest.raises(PermissionDenied):
            issue_refund("order-1", amount, approved_limit=100)

These tests are fast, free and fully repeatable. They are also your strongest guarantee, because rules enforced in code do not depend on the model's cooperation.

Layer 2: script the model to test control flow

Next, replace the model with a fake that returns predetermined replies. This lets you test the behaviour of the agent loop without cost, delay or randomness, including situations that are rare in real use: a tool error, a malformed reply, a model that never stops calling tools.

# Illustrative only. run_agent accepts its model as a dependency.
class ScriptedModel:
    def __init__(self, replies):
        self.replies = iter(replies)

    def complete(self, messages, tools):
        return next(self.replies)

def test_agent_stops_at_step_limit():
    looping = [tool_call("search", {"query": "x"})] * 20
    result = run_agent("Find x", model=ScriptedModel(looping), max_steps=5)

    assert result.status == "step_limit_reached"
    assert len(result.tool_calls) == 5

def test_tool_error_is_returned_to_the_model():
    replies = [tool_call("search", {"query": "x"}), final_text("Search failed.")]
    tools = {"search": failing_tool(TimeoutError())}

    result = run_agent("Find x", model=ScriptedModel(replies), tools=tools)

    assert result.status == "completed"
    assert "error" in result.messages[-2]["content"].lower()

For this to work, the model must be injectable rather than created deep inside the agent. Designing for that seam is the most useful testing decision you can make early, and it costs almost nothing.

Replay extends the idea. Record the model's responses from a real run, then play them back. It is useful for capturing a production failure as a permanent regression test. Remember that a replay proves your code still handles those responses. It says nothing about what the model would do today, so a recording can hide a behaviour change, and needs refreshing when prompts or tool definitions change.

Layer 3: run scenarios against the real model

Deterministic tests cannot tell you whether the agent actually chooses well. For that, write scenarios: a task, a controlled environment and a set of checks. Use a sandboxed or fake backend, so that the agent's actions are real to it but harmless, and so that you can inspect the final state.

Assert on behaviour and outcome, not words

Checks fall into several kinds, listed roughly from the most to the least dependable:

• Final state. Was the ticket created with the correct priority? Was the order left untouched? This is often the most meaningful check, because it allows for different routes to the same result.
• Forbidden actions. Did it avoid calling a delete tool? Did it stop for approval before a risky step? These are often strict requirements, so one failure may be enough to fail the scenario.
• Required tool calls. Did it check the stock before promising delivery? Avoid demanding one exact sequence unless the order truly matters, or you will fail valid plans.
• Resource budgets. Did it finish within reasonable step, token and time limits? A correct answer after thirty tool calls still points to a problem.
• Answer properties. Does it contain the order number? Is the output valid against a schema? Does it avoid a competitor's name?
• Judged quality. For qualities that code cannot check, such as helpfulness, use a second model as a grader with a written rubric. Check the grader against examples labelled by people, because it can be wrong, and calibrate it before you trust its scores.

Run each scenario several times

One pass tells you little about a non-deterministic system. Repeat each scenario and record a pass rate:

# Illustrative only. Run a scenario repeatedly and keep every failure.
def run_scenario(scenario, attempts=10):
    failures = []

    for attempt in range(attempts):
        environment = scenario.fresh_environment()
        result = run_agent(scenario.task, tools=environment.tools)
        problems = [check.name for check in scenario.checks
                    if not check.passes(result, environment)]
        if problems:
            failures.append({"attempt": attempt, "problems": problems,
                             "trace_id": result.trace_id})

    return {"pass_rate": 1 - len(failures) / attempts, "failures": failures}

The number of attempts and any thresholds here are placeholders. Small samples are noisy: a change from nine passes out of ten to eight out of ten could easily be chance, so do not treat it as a regression without more runs. Choose attempt counts according to how much the scenario matters and how much it costs, and set thresholds for each scenario. A requirement such as "never issue an unapproved refund" should tolerate no failures at all, while a preference about tone can tolerate some.

Always keep the trace of a failure. A pass rate tells you that something is wrong; a trace tells you what. The guide on LLM observability and tracing covers what to record. Failures in retrieval deserve separate scoring too, and how to evaluate a RAG system explains how.

Include awkward scenarios

• Impossible tasks. Does it admit that it cannot do something, or does it invent a result?
• Missing information. Does it ask a question or guess?
• Tool failures. Does it retry sensibly or claim success? An agent stating that it completed something that failed is a particularly damaging error.
• Injection attempts. Place hostile instructions in tool results and documents, and check that protected actions stay blocked. See LLM guardrails and prompt injection defence.
• Approval paths. Check that execution pauses and that a rejection is respected, as covered in human-in-the-loop approval.
• Long conversations. Does the agent still honour an early constraint after many turns?

Reducing noise without hiding the truth

You can make runs more repeatable by fixing model versions, freezing tool data and clocks, and setting a low temperature. Some providers offer a seed parameter, though check its current status and do not assume it gives identical results. These steps help you to compare two versions fairly. But they shrink rather than remove the variation, and a test environment that is too tidy may miss what real data does to the agent. Keep some scenarios on realistic, messy inputs.

Wiring it into your workflow

1. Treat prompts, tool descriptions and model versions as code. Version them, because a change to any of them can change behaviour.
2. Run layers 1 and 2 on every commit. They need no API key and should block merges when they fail.
3. Run scenarios when behaviour might change, and compare the results with a stored baseline from the current version rather than against a fixed number alone.
4. Turn production failures into tests. Each real incident should become a scenario or a replay, so that the suite grows from things that truly went wrong.
5. Control the cost. Scenario runs use real tokens. Keep a small critical set for frequent runs and a larger set for scheduled ones.

Offline tests cannot anticipate everything users will do, so keep monitoring after release. Roll out significant changes gradually where you can, and watch tool error rates, step counts and escalations alongside user feedback.

Frameworks that represent an agent as an explicit graph can also help, since individual nodes and routes can be tested in isolation. If you want help building an agent you can test this way, the orchestration with LangGraph page describes what I can help with.

Frequently asked questions

Can AI agents be tested with normal unit tests?

Yes, for a large part of the system. Tools, parsing, permissions, limits and the agent loop can be tested deterministically by replacing the model with a scripted fake. Only the model's choices need repeated, statistical evaluation against the real model.

How many times should I run each agent test?

There is no universal number. It depends on how critical the scenario is, how much the runs cost and how small a change you need to detect. More runs give firmer conclusions, and small differences from a handful of runs may be chance.

Should agent tests check the exact sequence of tool calls?

Usually not. Several sequences can be valid, so strict checks fail good plans. Prefer to check the final state, forbidden actions and any calls that are truly required, and enforce an order only where the task really depends on it.

Does setting temperature to zero make an agent deterministic?

Not reliably. It can reduce variation, but it does not guarantee identical output, and changes to models, tools or data can still alter behaviour. Treat it as a way to reduce noise when comparing versions, not as a substitute for repeated runs.