Fine-Tuning LLMs: When and How to Do It

When fine-tuning is justified

Fine-tuning continues training from an existing model using your examples, which changes how the model behaves. If you are still working out whether your problem is one of knowledge or of behaviour, read RAG vs fine-tuning first, because that guide makes the comparison. This one assumes that you have a behaviour problem, and it deals with what comes next.

Before you commit, check that you can answer yes to these questions:

Is the task narrow and repeated? Labelling tickets into fixed categories gives a model one clear pattern to learn. "Be a better assistant" does not.
Have you exhausted the prompt? A clearer prompt, some examples and structured output can solve the problem with no training at all. See prompt engineering for production LLM apps.
Do people agree on the correct output? If two careful reviewers often disagree, a model will learn the confusion. Fix the task definition first.
Do you have a baseline? Without measurements from your current approach, there is no way to tell whether training helped.
Can you maintain it? Requirements change and base models are retired, so you will probably repeat the work.

Check where you can train

This has become an important early question, as hosted options have been changing. At the time of writing, OpenAI's model optimisation guide says that it is winding down its fine-tuning platform and that the platform is no longer open to new users, although existing tuned models stay available until their base models are deprecated. Google's documentation says that no model in the Gemini API or AI Studio currently supports fine-tuning, and it points to its enterprise agent platform instead.

Other routes include tuning an open-weight model with a library such as Hugging Face's TRL. That route gives you control, but also the responsibility for hardware, hosting and monitoring. Before you prepare any data, confirm that your chosen base model can be tuned, that its licence suits your use, and that you know how long it will be supported. The LLM model selection guide covers the wider comparison.

Step 1: define the task and collect examples

Write down the input, the exact output you want and the rules for difficult cases. For a classifier, that means a fixed label list with a definition for each label, and a decision on what to do with inputs that fit none of them.

Real data is usually better than invented data, because it contains the mess the model will meet in use. Historical records have their own problems, however. Old labels may reflect outdated rules, or the habits of whoever handled the ticket. They may also contain personal data, which should be removed or masked unless you have a clear basis for using it, as a model can reproduce text from its training data. Include some inputs where the correct behaviour is to decline or to say that the information is missing, or the model will have no examples to learn that from.

Review a sample before going further. Have two people label the same small batch independently, and discuss any disagreements. This is often the most informative hour of the project.

Step 2: clean and split the data

This illustrative sample removes invalid labels and normalised duplicates, then splits by group. It uses only the Python standard library:

# Illustrative: cleans labelled tickets and splits them by customer.
import hashlib
import json
import random


def normalise(text):
    return " ".join(text.split()).casefold()


def clean(rows, allowed_labels):
    seen, kept, rejected = set(), [], []
    for row in rows:
        key = normalise(row["text"])
        if row["label"] not in allowed_labels or key in seen:
            rejected.append(row)
            continue
        seen.add(key)
        kept.append(row)
    return kept, rejected


def split_by_group(rows, group_key, test_share=0.2, seed=7):
    groups = sorted({row[group_key] for row in rows})
    random.Random(seed).shuffle(groups)
    test_groups = set(groups[:max(1, int(len(groups) * test_share))])
    train = [r for r in rows if r[group_key] not in test_groups]
    test = [r for r in rows if r[group_key] in test_groups]
    return train, test

The splitting choice deserves the most attention. A purely random split can place very similar tickets from the same customer on both sides, so that the test partly measures memory. Splitting by a group such as customer, document or conversation gives a more honest estimate. If your data changes over time, consider training on older examples and testing on newer ones. This better reflects how the model will be used.

Keep three sets if you can: training data, a validation set for choosing between runs, and a test set that you open only for the final comparison. Each time you adjust something after viewing test results, the test becomes a little less trustworthy. The sample shows just one split, and you can apply it again to carve out validation data.

Look at the rejected list as well. A high count of invalid labels usually points to a process problem, and that is worth fixing at its source. Also check the balance across labels, because a rare but important category can be swamped. If you add more of it, do so in training data only, and keep the test set representative of reality.

Step 3: format the examples

Trainers expect specific formats. TRL, for example, accepts standard and conversational data, in language modelling or prompt-completion form, and it applies the model's chat template automatically to conversational data. This illustrative sample writes a conversational JSONL file and records a fingerprint:

# Illustrative: writes the conversational format; check your trainer's.
SYSTEM = "Label the support ticket with exactly one category."


def to_example(row):
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": row["text"]},
        {"role": "assistant", "content": row["label"]},
    ]}


def write_jsonl(rows, path):
    with open(path, "w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(json.dumps(to_example(row), ensure_ascii=False) + "\n")


def fingerprint(path):
    with open(path, "rb") as handle:
        return hashlib.sha256(handle.read()).hexdigest()

Each example should look like a real request. That means the same system prompt, the same input format and the same surrounding context that you intend to use in production. If the live system will supply retrieved passages, then the training examples should include them as well.

Check the output after formatting by rendering a few examples through the chat template and reading the result. Template and end-of-sequence problems can be easy to miss. TRL's documentation specifically warns that the EOS token must align with the chat template for responses to terminate correctly. Check for truncation too, because a long example silently cut off will teach the model something you did not intend. The fingerprint links each trained model to the exact file that produced it.

Step 4: start small

Train first on a subset, with few settings changed. A quick run shows whether the pipeline works and whether the model is moving in the right direction. Record the base model and its version, the data fingerprint, the settings and the prompt for every run, so that any result can be reproduced.

When tuning an open-weight model, adapters are often a practical starting point. TRL integrates with the PEFT library for training an adapter rather than the entire model. Watch training and validation metrics together. If training loss keeps falling while validation results get worse, the model is probably overfitting. Adding more varied examples, or stopping earlier, is a better response than simply training for longer.

Step 5: evaluate against a baseline

The important question is not whether the tuned model looks good, but whether it beats the best alternative on data it has not seen. The baseline should be your best prompted model, not a neglected one. This illustrative sample compares both, label by label:

# Illustrative: compares two models per label on the same held-out set.
from collections import Counter


def per_label_recall(cases, predict):
    totals, correct = Counter(), Counter()
    for case in cases:
        totals[case["label"]] += 1
        if predict(case["text"]) == case["label"]:
            correct[case["label"]] += 1
    return {label: correct[label] / totals[label] for label in totals}


def compare(cases, baseline, tuned):
    before = per_label_recall(cases, baseline)
    after = per_label_recall(cases, tuned)
    return {
        label: {"baseline": before[label], "tuned": after[label]}
        for label in sorted(before)
    }

Here baseline and tuned are functions that you provide, each taking text and returning a label. Per-label results matter because an overall score can hide a failure in a small, high-stakes category. Recall alone is not the whole picture either, so extend this with precision for any label where false positives carry a cost. For free-text output, use a written rubric and reviewers who cannot see which model produced which answer, along with a check for factual errors.

Look beyond the headline too:

Size of the difference. On a small test set, a gap of a few cases may be noise. Read the disagreements between the two models.
New failures. Note what the tuned model now gets wrong that the baseline got right.
Safety behaviour. Retest refusals and adversarial inputs, because training on narrow examples can change general behaviour.
Running cost and latency. Measure these with current prices or your own hardware figures.
Stability. Train again with a different seed if you can, to see how much results vary.

Decide in advance what result would justify the switch. If the tuned model does not clear that bar, keep the simpler system. That is a useful result, not a failure.

Step 6: ship and keep watching

Roll out to a small share of traffic, with the baseline available as a fallback, and version the model alongside its prompt. Log inputs and outputs in line with your privacy rules, and sample them for review. LLM observability and tracing covers what to capture. Corrected production errors can become new training and test examples, but keep the same split discipline when adding them. Set a trigger for retraining too, such as a change to the label set or a drop in reviewed quality, as a tuned model will not adapt on its own.

If you are weighing up a fine-tuning project, the generative AI and LLM development page describes what I can help with.

Frequently asked questions

How many examples do I need to fine-tune an LLM?

There is no universal number. It depends on the task, the variety of inputs and the base model. Start with a modest set, measure on held-out data, and add examples for the cases that fail. A provider may also set its own minimum.

Can I still fine-tune models from OpenAI or Google?

Availability has been shrinking, so check first. At the time of writing, OpenAI says it is winding down its fine-tuning platform and has closed it to new users, and Google says the Gemini API has no tunable model.

What is data leakage in fine-tuning?

Leakage occurs when test examples, or near copies of them, also appear in training. The model then appears better than it is. Removing duplicates and splitting by customer, document or time period reduces the risk.

Is a lower training loss proof that the tuned model is better?

No. Loss shows how the model fits its training objective, and it can keep falling while performance on new data stalls or declines. Judge the model by task measures on a held-out test set.

Fine-tuning LLMs: when and how