How to scope an AI project before you build

Why scoping matters more with AI

Ordinary software is often specified by what it should do: when a user clicks this, save that. AI features are harder to pin down, as their output varies and will sometimes be wrong. You cannot fully specify the behaviour in advance, so you need to agree how good is good enough, how that will be measured and what happens when the system fails.

A demonstration can also make a task look finished long before it is. A model that answers ten hand-picked questions well says little about how it will handle unusual requests, messy documents or users who should not see certain data. Good scoping brings those questions forward, when changing direction is still easy. You do not need to know how the technology works to answer most of them, because they concern your business.

Decision 1: what is the problem, in business terms?

Begin with the problem and not the technology. "We want a chatbot" describes a possible solution. "Our support team spends much of its time finding answers scattered across policy documents" describes a problem and leaves the solution open.

Try to complete this sentence: [This group of people] currently [does this], which causes [this problem], and we would know it had improved if [this changed]. If people in your organisation complete it differently, resolve that first, as no supplier can settle it on your behalf.

Decision 2: how is it handled today?

Record the current process before proposing a new one. Who does the work, what do they consult, what does a good result look like and where does it go wrong? Then take some measurements of your own, even rough ones, such as how often the task occurs and how long it takes.

This baseline does two jobs. It gives you something real to compare against later, and it often reveals useful details, such as an expert's unwritten rule or a step that exists only because of an old system. Without a baseline, it is difficult to show that a new system has improved anything.

Decision 3: does this need AI at all?

It is worth asking this honestly, and a good engineer should be willing to raise it. Fixed rules, a better search box, a form or a report may solve the problem more cheaply and predictably. AI tends to fit where the input is unstructured, the variety defeats fixed rules and some errors can be tolerated or caught.

Even then, the AI part is often small. A workflow may use a model to classify a message and ordinary code for everything else. Similarly, you may not need several cooperating agents; single agent vs multi-agent systems sets out when added complexity is justified. Choosing the least complicated design that works is a sign of good judgement, not a lack of ambition.

Decision 4: what data can the system use?

Data problems are a common cause of trouble, and they are easier to find before work starts. Establish the following:

• Where it lives. List the sources and who controls each one.
• Whether it can be accessed. Is there an export or an API? Does someone need to approve access?
• Its condition. Is it current, consistent and complete? Are there scanned files or conflicting versions? Look at a real sample rather than a description.
• Whether you may use it. Check contracts, personal data rules, sector rules and whether the data may be sent to an external model provider. Involve your legal or compliance contact where necessary.
• Who may see what. If existing documents have access restrictions, the AI system must respect them.

Share a representative sample early, including the awkward cases. A supplier given only the tidiest documents will form an unrealistic view of the work. If your content includes difficult files, RAG for PDFs and tables shows why this can be a substantial piece of work in its own right.

Decision 5: what errors can you accept?

Every AI system makes mistakes, so the useful question is which mistakes matter and how much. Not all errors are equal. A poor draft that an employee corrects is a minor cost. A wrong statement sent straight to a customer about a refund is another matter. Work through these questions:

• What is the worst realistic outcome of a wrong answer?
• Is it worse to give a wrong answer or to give no answer?
• Can someone tell that an output is wrong? How easily?
• Can the action be undone?

Your answers largely determine how much autonomy is reasonable. They may also rule out some ideas altogether, and it is better to discover this now.

Decision 6: where do people stay involved?

Decide the level of autonomy as a deliberate choice. The system could suggest while a person decides, act with approval, act and be spot-checked, or act alone on low-risk tasks. Starting with more oversight and relaxing it as evidence builds is often a sensible path.

Be specific about the practicalities. Who reviews the output, what do they need to see to judge it and what happens when they are unavailable? A review step that nobody has time for offers little real protection. Human-in-the-loop approval for AI agents looks at the design of this step, including approval fatigue.

Decision 7: how will you judge success?

Agree a test before the build begins. Ask your domain experts to collect realistic examples with good answers, including hard cases and some that the system should decline. This set becomes the shared definition of "working", and it is much harder to argue with than an impressive demonstration.

Use two levels of measure. Quality measures ask whether the outputs are correct, supported and in the right format. Business measures ask whether the process really improved, for instance through less handling time or fewer escalations. Set your targets using your baseline and the cost of errors, rather than borrowing a figure from elsewhere. For a document assistant, how to evaluate a RAG system shows what this looks like in practice.

Decision 8: how does it fit into daily work?

A capable system that sits outside people's working tools may go unused. Decide where it will appear, such as an existing application, a helpdesk or a new interface. Then list what it must connect to, how users will sign in, what must be logged and how much delay users will accept. Include IT and security contacts early, since their requirements can shape the design. The guide to adding an LLM feature to a full stack app shows the sort of concerns involved.

Decision 9: what is left out, and who owns it afterwards?

Write down what the first version will not do. Choose one user group, one or two content sources and one task, then record the other good ideas for later. A narrow release gives you real evidence, while a broad one can spread attention thinly.

Also plan for life after launch, because an AI system needs ongoing care. Documents change, model versions are retired and new questions arise. Decide who updates the content, who reviews failures, who pays and monitors usage costs, and who may approve changes. Ask for the code, prompts, test sets and configuration to be handed over in a form your team can use.

Cost and duration should come out of this work and not precede it. A credible estimate depends on answers to the questions above, so be wary of a firm figure given before anyone has looked at your data.

Questions to ask a supplier

• Might a simpler, non-AI approach work here?
• What would you want to test first, and what result would make you recommend stopping?
• How will quality be measured, and will we see the failures as well as the successes?
• Where will our data go, and what is stored?
• What are the ongoing running costs driven by, and how can we control them?
• What will we own and be able to maintain when the work ends?

Answers that acknowledge uncertainty and propose a way to reduce it are generally more reassuring than confident promises.

Turning this into a brief

A useful brief can be short. Cover the problem, current process, users, data sources and sample, unacceptable errors, level of oversight, success test, first-version boundaries and owner. Gaps are fine as long as they are marked as open questions. If you are preparing to commission this kind of work, the generative AI and LLM development page describes what I can help with.

Frequently asked questions

Do I need a technical specification before talking to an AI engineer?

No. A clear description of the problem, the current process, the available data and what a good result looks like is more useful at first. The technical design can be worked out from those answers.

What is the difference between a proof of concept and a production system?

A proof of concept tests whether an approach can work, often on limited data. A production system must also handle permissions, failures, monitoring, unusual inputs and ongoing maintenance, which are often substantial additional work.

Can a supplier guarantee a level of AI accuracy?

Be cautious of an accuracy promise that is not tied to a defined test. Performance depends on the data, the task and how it is measured. A more useful commitment is an agreed evaluation method and reported results.

What if our data is not ready?

Treat it as a finding and not a failure. You might narrow the first version to a cleaner source, or plan a data preparation phase. Testing a representative sample early shows what needs to be done.