AI agents for data analysis: safe uses and how to check results

Think in levels of consequence

"Data analysis" covers tasks with very different risks. Summarising the columns in a file does little harm if it goes wrong. A revenue figure repeated to a board can do a lot. Sorting an agent's work by what happens when it is wrong gives a better guide to its permissions than any general view on whether agents can be trusted.

Most of the value sits in the first three levels. An agent that gets an analyst to a checked first draft sooner is useful even if it never concludes anything by itself.

What an agent does well

• First-pass profiling. Column types, missing values, duplicates and odd ranges are tedious to check and easy to verify, which makes them a natural starting point.
• Drafting analysis code. Generating a first version of a pandas script or chart is quick, and because the output is code, a person can read it and run it again.
• Trying routine variations. Repeating an analysis by region, month or product is mechanical work that a person may skip when short of time.
• Explaining a table in plain language. Provided the numbers come from computed output, a model can describe them for a reader who does not work with data.

Agents are weaker where the answer depends on knowledge the data does not contain: that a drop in March reflects a tracking change, that one customer is really a test account, or that a field stopped being filled in after a system migration. A confident explanation of such a pattern can be entirely invented, so causes need a person who knows the business.

Contain the code it runs

An analysis agent often writes and runs Python, so it needs an execution environment you would be comfortable giving to unreviewed code. When the agent queries a database directly, the separate guide on text-to-SQL risks and safeguards covers that surface, and it applies alongside what follows. For code execution, the essentials are these:

• Work on a copy. Mount an extract or snapshot read-only, so that a bad script cannot alter the source.
• Isolate the runtime. Use a disposable container or similar sandbox with no credentials inside it and no network access unless a specific destination is required. A file can contain instructions as well as data, and without a network route a manipulated script has a much harder time sending anything out.
• Limit resources. Cap time, memory and output size, and record when output was cut short.
• Minimise the data. Remove columns the question does not need. If a hosted model will see results, check that your privacy obligations and provider terms allow it, and consider sending aggregates in place of rows.
• Keep every artefact. Save the code, the input version, the outputs and the final narrative together, so that anyone can reproduce a result later.

How to check the results

Code that runs without an error has only shown that it runs. The mistakes that matter in analysis usually produce plausible figures, so each check below looks for a specific way of being wrong.

Reconcile with something you already trust

Before accepting a breakdown, confirm that its parts add up to a total from an independent source, such as a finance report or a figure already signed off. Suppose an agent splits last quarter's sales by region, and the regions sum to noticeably more than the known company total. That points to double counting somewhere, however neat the table looks. Agree on these control totals in advance so that the check does not depend on what the agent happened to report.

Watch row counts through each step

Joins are a common source of silent error. If one order matches two rows in a poorly keyed customer table, the order is counted twice and every downstream sum inflates. Have the agent's code record row counts before and after each join and filter, and fail when a count changes in a way the step should not allow.

# Illustrative only: a guard an analysis script can call after a join.
# The DataFrames and the key name are placeholders.
def check_join(before, after, key):
    if len(after) != len(before):
        raise ValueError(
            f"Row count changed from {len(before)} to {len(after)}"
        )
    if after[key].duplicated().any():
        raise ValueError(f"Duplicate values in {key} after the join")
    return after

This sample is illustrative and assumes a join that should keep exactly one row per key. A deliberate one-to-many join needs a different rule, which is part of why the expected result should be stated before the code runs.

Interrogate nulls and denominators

Missing values change answers without raising errors. In PostgreSQL, for example, count(column) counts only rows where that column is not null, whereas count(*) counts every row, so an average or a rate can be built on a smaller population than the reader assumes. Ask the agent to state, for every percentage, what the denominator is, how many rows it covers and how many were excluded.

Trace every number in the narrative

A language model can restate a figure wrongly, round it oddly or produce one that was never calculated. A simple code check catches a good share of these slips: extract each number from the written summary and flag any that do not appear in the computed results.

# Illustrative only: flags narrative numbers absent from computed output.
# It will not catch a real number attached to the wrong label.
import re

NUMBER = re.compile(r"\d[\d,]*(?:\.\d+)?")

def untraced_numbers(summary, computed_values):
    known = {round(float(value), 2) for value in computed_values}
    flagged = []
    for match in NUMBER.findall(summary):
        value = round(float(match.replace(",", "")), 2)
        if value not in known:
            flagged.append(match)
    return flagged

Treat a flag as a prompt to look, because derived values, dates and rounded figures will also trigger it. Better still, generate the figures in the report from the results directly, and let the model write only the sentences around them.

Recompute independently

For a figure that matters, calculate it a second way: a different tool, a hand-written query, or a second person starting only from the metric's definition. Agreement between two separately built calculations is much stronger evidence than a review of one. Asking the same agent to check its own answer is weaker, since it may repeat its original assumption.

Test the conclusion, not only the arithmetic

Correct arithmetic can still support a poor conclusion. These questions are worth asking of any finding an agent presents:

• Does it hold within groups? Under Simpson's paradox, a pattern seen in separate groups can vanish or reverse when they are combined, so try the obvious segments.
• How many cuts were tried? An agent can test dozens of segments in seconds, and some will look striking by chance. Ask for the full list tried, not only the winner.
• How sensitive is it? Change the date window slightly or remove the largest customer. A conclusion that flips deserves much less confidence.
• Does it claim a cause? Observational data can show that two things moved together. It rarely shows why, and the wording should say so.

Measure the agent with planted problems

Run the agent against test data whose answers you already know. Then seed that data with defects a careful analyst would notice: a duplicated key, a block of missing dates, a mislabelled unit, or an outlier from a data entry error. Score whether it reports each defect as well as whether the final figure is right. This reveals something an accuracy check on clean data cannot, namely whether the agent notices when it should stop and raise a concern. Keep the set under version control and re-run it when the model, prompt or tools change. Testing AI agents covers the wider practice, including how to deal with varying output.

Make review easy

A reviewer can only challenge what they can see. Present results with the question as interpreted, the data version and date range, any exclusions and assumptions, the code, and the outcome of each automated check. For findings that feed a decision, route them through a defined sign-off, using the patterns in human-in-the-loop approval for AI agents.

Whether an agent is the right tool at all depends on the question. Figures over structured data call for computation, whereas answers written in documents do not, a distinction set out in SQL agents vs RAG.

If you are planning an analysis workflow with several steps or specialised agents, the multi-agent systems page describes what I can help with.

Frequently asked questions

Can I give an AI agent direct access to production data?

It is better not to. Give it a read-only snapshot or a restricted view with only the columns it needs, and run its code in an isolated environment. This limits the damage from both mistakes and manipulated input.

How do I know an AI agent's analysis is correct?

Reconcile its figures with trusted totals, inspect row counts through joins and filters, confirm that each number in the write-up came from computed output, and recalculate important results independently. For decisions, have someone who knows the data review the conclusion.

Can an AI agent replace a data analyst?

It can speed up profiling, drafting code and routine variations, but it lacks context the data does not hold, such as tracking changes or business events. Treat it as a way to reach a checkable draft sooner, with a person accountable for conclusions.

Should the agent check its own work?

Self-checks can catch simple slips, but an agent may repeat the assumption that caused the error. Automated checks written in advance and an independent recalculation provide much stronger evidence.