How to choose an LLM by testing it against your task

Why general rankings do not settle it

Benchmarks and leaderboards measure performance on someone else's tasks, under someone else's conditions. A model that leads on coding puzzles tells you little about how it will extract clauses from your contracts, or handle support tickets in Hindi and English mixed together. Rankings also move quickly, which is why this guide names no vendors or prices: any list here would date, and it would answer the wrong question.

The durable skill is a repeatable selection process. Once you have it, a new model release becomes a routine test run rather than a debate.

Step 1: define the job precisely

"We need a good model" cannot be tested. Write down what goes in, what must come out and what a failure costs. A useful description might read: "Given a customer email of up to a page, return one of nine categories and a one-line reason. A wrong category sends the ticket to the wrong team, which is annoying but recoverable."

Also separate the steps. A pipeline that classifies, retrieves and drafts contains several jobs, and they need not share a model. Selecting per step is often more useful than searching for one model that does everything. For the routing mechanics, see controlling LLM cost and latency.

Step 2: apply hard constraints first

Some requirements are pass or fail, and checking them first saves testing effort. Read the current documentation and terms for each item, as these details change:

• Data handling. Where is data processed, how long is it retained and may it be used for training? Does this meet your contractual and legal obligations?
• Deployment. Must it run in your cloud, on your own hardware or offline? This may decide between hosted APIs and open-weight models.
• Licence. For open-weight models, read the actual licence and any restrictions on commercial use rather than assuming "open" means unrestricted.
• Required features. Tool calling, structured output, image input or a particular language may be essential.
• Context limits. Check both the input window and the maximum output length against your largest realistic request.
• Rate limits. Compare your account's limits with your expected peak traffic.

Whatever survives forms your longlist. Narrow it to three or four candidates of different sizes, including at least one smaller option. Testing only top-tier models means you never learn whether a cheaper one would have been enough.

Step 3: build a test set from real work

This is the most valuable part of the process, and it outlasts any one decision. Collect real inputs where you can, with personal data removed or handled in line with your obligations. Aim for coverage rather than a particular size:

• Ordinary cases in roughly the proportion you expect them.
• Hard cases that an expert would need to think about.
• Edge cases, such as very long, very short, badly formatted or mixed-language inputs.
• Cases where the correct behaviour is to refuse or say the answer is unknown.
• Past failures, which should stay in the set permanently.

A few dozen well-chosen cases are often enough to eliminate a poor candidate. Separating two close finishers takes more, because a small difference on a small set may just be noise. Write down what a correct answer looks like before you see any model output, otherwise you risk grading in favour of whichever style you happen to like. For retrieval pipelines, the same discipline applies with extra measures, set out in how to evaluate a RAG system.

Step 4: decide how to score

Use the most objective scoring the task allows:

When people review generated text, hide which model produced each answer and shuffle the order. Knowing the name of a model invites bias, however well-meaning the reviewer. If you use an LLM as a judge, check its verdicts against a sample of human ones before trusting them.

Step 5: run a fair comparison

Give every candidate a reasonable chance, and record more than the answers. A simple harness is enough, as in this illustrative sketch:

# Illustrative only. call_model and score are placeholders for your code.
import statistics
import time

def evaluate(model, cases, repeats=3):
    results = []
    for case in cases:
        for _ in range(repeats):
            start = time.perf_counter()
            reply = call_model(model, case["input"])
            results.append({
                "case": case["id"],
                "passed": score(reply.text, case["expected"]),
                "seconds": time.perf_counter() - start,
                "tokens_in": reply.tokens_in,
                "tokens_out": reply.tokens_out,
            })
    return {
        "pass_rate": statistics.mean(r["passed"] for r in results),
        "median_seconds": statistics.median(r["seconds"] for r in results),
        "results": results,
    }

The repeat count is an arbitrary example. A few details affect whether the comparison is fair:

• Adapt the prompt modestly for each model. A prompt tuned for one may undersell another. Give each candidate a similar amount of tuning, using a separate development set, so you are not fitting the prompt to the test itself. Prompt engineering for production LLM apps covers how to manage those versions.
• Repeat runs. Outputs vary between calls, even at low temperature settings, so a single run can flatter or punish a model.
• Measure from where you will deploy. Latency depends on region, load and time of day. Run tests at more than one time, and look at slow requests as well as the median.
• Use streaming measures where relevant. In a chat product, time to first token may matter as much as total time. Record both.

Step 6: compare on cost per successful result

Price per token is a misleading basis for comparison. What matters is the cost of getting an acceptable result, including what failure costs you. A cheaper model that needs retries, longer prompts or human correction may end up more expensive overall. Calculate it from your recorded token counts and each provider's current price list, then add the cost of handling failures.

Bring the results together in one table and decide using a rule you set in advance. A sensible one is this: of the models that clear your quality bar and your latency limit, choose the cheapest and simplest to operate. A higher score above the bar is only worth paying for if users or the business would actually notice. Also look at the failures themselves, not just the rates. Two models with similar scores may fail in very different ways, and a confident wrong answer usually does more damage than a refusal.

Step 7: keep the model replaceable

Your choice has a limited shelf life. Models get deprecated, prices change and better options appear, so design for replacement:

• Put model calls behind a small interface of your own, and keep model names in configuration.
• Store prompts per model, so a change can be tracked and rolled back.
• Pin to a dated or versioned model identifier where one is offered, since an alias may change its behaviour under you.
• Run the test set in your deployment pipeline, and whenever you consider a change.
• Check that features you rely on, such as tool calling or structured output, behave the same way on an alternative.

Before a full switch, consider running the new model on a copy of live traffic without showing its output, then releasing it to a small share of users while monitoring it. LLM observability and tracing covers what to record. An offline test set never fully matches production, so keep adding interesting real failures to it.

Should you self-host?

Self-hosting an open-weight model is a decision about operations as much as about models. It can address data residency, offline use or control over versions. In return you take on hardware, scaling, updates and monitoring. Do not assume it will be cheaper, as that depends on your volume and on how fully the hardware is used. Evaluate the model for quality first, using the same test set, and then work out the running costs from your own traffic.

If you would like help setting up a test harness or designing a model layer, see generative AI and LLM development.

Frequently asked questions

What is the best LLM for my application?

No single model is best across all tasks. The right one is the simplest model that passes your own tests within your data, latency and budget constraints. Testing a short list on real examples will tell you more than any general ranking.

How many test cases do I need to compare LLMs?

It depends on how close the candidates are and how varied the task is. A few dozen well-chosen cases can eliminate a clearly weak option, while separating close finishers needs more. Coverage of hard and edge cases matters more than a target number.

Are LLM benchmarks useless?

No. They are useful for spotting candidates and for following broad changes in capability. They are weak evidence about your task, though, because the data, the prompts and the scoring all differ from yours, so treat them as a starting point.

How often should I re-evaluate my model choice?

Re-run your test set when a provider announces a deprecation, when a credible new option appears, when costs or volumes change substantially or when production quality slips. An automated test set makes these checks routine rather than a project.