AI Chatbots for Customer Support: A Buyer's Guide

Start from your tickets, not from the technology

The most useful first step costs little: read a sample of recent support conversations and sort them by what the customer needed. Vendor demonstrations tend to show the bot at its best. Your tickets show the questions it will really face, in the words customers really use, including the misspelt, angry and ambiguous ones.

For each group of requests, note how often it appears, where the correct answer lives, what could go wrong if the answer were wrong, and whether a person needs to make a judgement. This gives you an evidence-based shortlist. It also tends to expose problems no chatbot can fix, such as a policy that is written down nowhere, or two documents that disagree with each other.

What to automate, and what to leave alone

It helps to think of support automation in three tiers, each carrying more risk than the last.

Tier	What the bot does	Example	What it needs
Answer	Explains something from approved content	"What is your returns window?"	Current, consistent documentation
Look up	Reads a customer's own record	"Where is my order?"	Identity checks and read-only system access
Act	Changes something on the customer's behalf	"Cancel my subscription"	Strict rules, audit logs and confirmation steps

Most projects are safer if they prove the first tier before moving to the next. A wrong answer is a problem that can be corrected. A wrong action, such as a refund issued to the wrong person, can be much harder to undo.

Good early candidates

Common questions with one correct answer in your documentation.
Status requests that can be answered by reading from a system.
Collecting details before a handover, so that the agent does not need to ask again.
Helping customers find the right article or form.

Keep with people, at least at first

Complaints, cancellations made in frustration and anything involving vulnerable customers.
Legal, medical, financial or safety questions.
Exceptions to policy, in which the correct answer depends on discretion.
Decisions about money or access, unless they follow fixed rules enforced in code.

Where the answers come from

A support bot built on a general model alone may produce a confident answer about a policy it has never seen. The usual remedy is to retrieve passages from your approved content and have the model answer from them, a pattern explained in what RAG is and when to use it. Live details such as order status should come from your systems through tool calls, not from an index that may lag behind.

This makes content quality a project dependency. If your help centre is out of date, the bot will repeat outdated information politely. Ask who owns each source, how an update reaches the bot and how a retired policy is removed. Before you compare suppliers, also consider whether a hosted assistant would be enough, as discussed in RAG chatbot vs custom GPT.

Designing the escalation path

Escalation is a core part of the product, and it deserves the same attention as the answers do. A bot that traps customers in a loop can do more harm than having no bot at all.

When to hand over

The customer asks. A request for a person should be honoured promptly, without repeated attempts to keep the conversation with the bot.
The topic is on a restricted list. Your policy decides this, and ordinary code can enforce it.
The bot has no support for an answer. Nothing relevant was found in the approved content.
The conversation is going in circles. Repeated rephrasing or failed attempts should trigger a handover.
The customer is distressed. Treat detection as imperfect, and provide a visible route to a person at all times.

Avoid relying on a model's own confidence as the only trigger, because a model can be wrong and still sound certain. Combine several signals, and keep the rules you can write down in code.

What a good handover contains

The person taking over should see the full transcript, a short summary, the details already collected, which sources the bot used and why it escalated. The customer should not need to repeat themselves. They should also be told what happens next, including what to expect outside your staffed hours. An escalation path that ends in an unattended inbox is not really an escalation path, so plan for capacity as well as for software.

For actions with real consequences, the same principle applies inside the workflow. Human-in-the-loop approval for AI agents explains how approval gates work.

Measuring quality

No single number will tell you whether a support bot is any good. Use a small set of measures that keep each other honest.

Correct resolution. Was the issue actually solved, and solved correctly? Sample and review conversations by hand to find out.
Answer accuracy and support. Is each answer right, and is it backed by the source material?
Repeat contact. Did the customer come back about the same issue soon afterwards? This may reveal false resolutions.
Escalation quality. Did the bot hand over when it should have, and did the handover contain what the agent needed?
Customer satisfaction, compared fairly. Compare similar kinds of request, since a bot may be handling the easier ones.
Harmful errors. Track serious mistakes, such as a wrong policy statement or exposed data, separately, and review every one.

Be careful with deflection or containment rates on their own. A customer who gives up and leaves can look identical to one who was helped. Agree on baselines from your current process before launch, otherwise you will have nothing to compare against. There are no universal targets worth borrowing, because the right threshold depends on your request types and the cost of an error.

Test before customers do

Build a test set from past tickets with personal data removed, and include questions the bot must decline, attempts to manipulate it, and requests that should escalate. Have people who know your policies mark the results. Then roll out to a limited share of conversations, review transcripts regularly and turn each serious failure into a new test case. How to evaluate a RAG system describes the technical side of this for retrieval-based bots.

Risks to raise with a supplier

A chat window is a public input to your systems. Customers, or attackers posing as customers, may try to extract information or provoke an unauthorised promise. Instructions in a prompt are not a security boundary, so permissions must be enforced outside the model. LLM guardrails and prompt injection defence covers this in more depth.

Ask any supplier, or your own team, these questions:

How is a customer's identity verified before account data is shown?
Can the bot reach any data or action beyond what that customer is allowed?
What is sent to third-party model providers, how long is it kept, and is it used for training?
Who can see transcripts, and how long are they retained?
How is a mistake traced to its cause, and how quickly can the bot be switched off?
Who maintains the content and test set after launch?

A sensible way to begin

Choose one group of requests from your ticket review, make sure its content is correct and decide on your measures and escalation rules before anything is built. Launch narrowly, review closely and expand only when the evidence supports it. How to scope an AI project turns these decisions into a brief.

If you would like help planning or building a support assistant, the generative AI and LLM development page describes what I can help with.

Frequently asked questions

Can an AI chatbot replace a customer support team?

Not as a sensible starting assumption. A chatbot can take on suitable repetitive requests, but complaints, exceptions and sensitive cases still need people, and someone has to maintain content and review the bot's performance.

What is the difference between deflection and resolution?

Deflection means a customer did not reach a person. Resolution means the issue was actually solved. A customer who gives up also counts as deflected, so deflection alone can make a poor chatbot look successful.

Should customers be told that they are talking to a bot?

Yes, in my view, because it sets fair expectations and helps people decide when to ask for a person. Disclosure rules vary by place and sector, so ask your legal adviser what applies to you.

How do we test a support chatbot before launch?

Replay realistic questions from past tickets with personal data removed, including out-of-scope, emotional and adversarial messages. Have people who know your policies score the answers, then run a limited rollout with reviewed transcripts and a working route to a person.

AI chatbots for customer support: a buyer's guide