Hire an AI Engineer: What to Look For

Why judging this is hard

It has become easy to produce an impressive AI demonstration. With a hosted model and a few hundred lines of code, anyone can build something that answers ten chosen questions beautifully. The hard part comes afterwards: odd documents, unusual questions, costs, security and the steady drift that follows a launch. A demonstration tells you almost nothing about that part, and it is the part you are paying for.

You do not need to be technical to judge this. Most of the useful signals concern how someone thinks about being wrong, and you can hear those in an ordinary conversation.

Work out what you need first

"AI engineer" covers quite different kinds of work. Be clear which of these you are buying before you start comparing people:

Application engineering. Building products on existing models: RAG, agents, integrations and interfaces. Many business projects fall here.
Machine learning engineering. Training, fine-tuning and serving models, usually when you have suitable data and a reason not to use an existing model.
Data engineering. Getting data clean and accessible, which is sometimes the real project.
Research. New methods with uncertain results. This is rarely what a business project requires.

A short written brief helps here. State the problem, who it affects, what data exists and what a wrong result would cost. How to scope an AI project gives you the questions. A brief also serves as a test: good candidates will challenge parts of it.

Freelancer, agency or employee?

A freelancer suits a well-bounded project, and you work directly with the person doing the work. Ask what happens if they are unavailable, and insist on handover documentation. An agency offers a broader mix of skills and more cover, so ask exactly who will do the work, because the person in the sales meeting may not be the one building it. An employee makes sense when AI work will be continuing and central. Whichever you choose, someone on your side must own the outcome.

What good evidence looks like

Treat these as stronger signals than a list of frameworks:

Work that can be inspected. Live projects, public code or detailed write-ups. Ask what they personally did, as opposed to the team.
A discussion of trade-offs. They can say why they chose one approach and what they gave up. A person who built it remembers the awkward bits.
Honest limits. They can describe where a project performed poorly, and they say "I don't know" without discomfort.
Measurement. They reach for a test set unprompted, and any figures they quote come with context: measured on what, and compared with what?
Software fundamentals. Testing, logging, error handling, security and deployment. An AI feature is still software, and much of the work is ordinary engineering.
Clear explanation. They can explain a technical choice to you without jargon, and they check that you followed it.

Note that a recent framework is a poor test. Tools in this field change quickly, and a capable engineer can pick up a new library. Depth in the problems, such as evaluation and retrieval quality, should matter more than the number of logos on a profile. Treat references, testimonials and quoted results as claims to check, and where you can, speak to a former client or colleague directly.

Questions to ask

These work for non-technical buyers, as you are listening mainly for specifics. Each one comes with what a stronger answer tends to include.

About approach

"Does this need AI at all?" Listen for a willingness to suggest something simpler, such as rules, search or a form. Someone who always recommends the largest build deserves caution. Compare single agent vs multi-agent systems if you are offered several agents without a clear reason.
"What would you want to see before giving an estimate?" A good answer asks for a sample of your real data, including the awkward cases.
"What is the riskiest assumption in this project, and how would you test it early?" Look for a concrete, cheap experiment, and not general reassurance.

About quality

"How will we both know that it works?" Expect a test set agreed with your domain experts, measures for each part of the system and a target tied to the cost of errors. For a document assistant, how to evaluate a RAG system shows what that looks like.
"What happens when the model gets it wrong?" They should talk about detection, fallback behaviour, human review and how users are told. "It won't" is the wrong answer.
"Tell me about a project that did not go as expected." Listen for specifics and for what changed afterwards. A person who has never had a project go wrong either lacks experience or lacks candour.
"How do you stop quality slipping when a prompt or model changes?" Look for automated regression tests, version control for prompts and monitoring in production. LLM observability and tracing gives the background.

About risk and data

"Where will our data go, and who can see it?" They should name every provider involved, ask about your obligations and discuss retention and logs.
"How do you make sure users only see what they are allowed to see?" The answer should involve enforcement in code or the data layer, not an instruction to the model.
"What could a malicious document or user do to this system?" They should know about prompt injection, and say that it is limited by restricting what the system can do. LLM guardrails and prompt injection defence explains the expected thinking.

About cost and ownership

"What will this cost to run, as well as to build?" Look for an explanation of what drives usage costs and how they can be monitored and capped.
"What happens when the provider retires this model?" Good answers cover how models can be swapped and how the test set validates a replacement.
"What will we own and receive at the end?" You want code, prompts, test sets, configuration, documentation and account access, all in your control.
"Who maintains this after launch?" Look for a realistic account of the ongoing work, and a plan your team could follow.

Warning signs

Guarantees of accuracy. Particularly a figure like a percentage promised before seeing your data.
A firm quote with no discovery. A credible estimate depends on having seen your data and agreed what success means.
No mention of evaluation. If testing means trying a few questions, you are likely to find the problems in production.
A solution looking for a problem. They recommend a framework, multiple agents or fine-tuning before they understand the task.
Vague claims about results. Unnamed clients and unexplained percentages that fall apart under a simple follow-up question.
Dismissing security or privacy. Being told "the model won't do that" or "the provider handles it" is not a plan.
No interest in your process. They do not ask how the work is done today or who would use the system.
Lock-in. Essential parts stay on their accounts, or the prompts and test sets are described as proprietary, with no agreed route for handover.

Run a small paid test

When the conversations go well, the strongest next step is a short, paid piece of work with a clear output, instead of committing to the whole project. Good options include a discovery study on a sample of your data or a narrow prototype with an agreed test set. Ask to receive:

The results on your own test cases, including those it failed.
A short written account of what was tried, what was learned and what is recommended next, even if that recommendation is to stop.
The code and instructions, so that someone else could run it.

Pay attention to the working relationship as well. Did they raise bad news early? Did they ask for clarification? Did the result match what they said it would be? A test like this reveals these things better than any interview, and it gives you a useful output even if you go no further.

What to put in the agreement

Take suitable legal advice, and at least cover the following: ownership of code and outputs, confidentiality and data handling, acceptance criteria based on the test set, who pays for model and hosting usage, the handover contents and what support is included after launch. Acceptance tied to an agreed test is far easier to apply than a phrase like "works well". If your use case is close to a specific pattern, reading a guide such as adding an LLM feature to a full stack app can also help you follow the conversation.

Applying this to me

The same questions apply to anyone you consider, and that includes me. To see the kinds of work I take on, look at the services page, then ask these questions in our first conversation.

Frequently asked questions

Do I need a technical person to help me hire?

It helps but is not essential. Non-technical buyers can judge a lot from how a candidate discusses errors, testing and risk. For a large or high-risk project, ask an independent technical person to review the proposal or the code.

Should I hire a freelancer or an agency for an AI project?

A freelancer can suit a bounded project where you want direct contact with the person building it. An agency offers wider cover and more skills. Either way, confirm who does the work and what is handed over.

How can I check an AI engineer's portfolio?

Ask them to walk you through a project: what they personally built, what they chose not to do, what failed and how quality was measured. Look at live work or code where you can, and speak to references directly.

Is a certification a good sign when hiring an AI engineer?

It may show interest and study, but it does not demonstrate that someone can make a system reliable on real data. Give more weight to inspectable work, clear reasoning about trade-offs and the results of a small paid test.

How to hire an AI engineer: what to look for and what to ask