Text Classification: LLMs vs Traditional NLP

What text classification involves

Text classification assigns one or more labels from a known set to a piece of text. Routing a support ticket, flagging a complaint, tagging a document by topic and detecting the language of a message are all classification tasks. The output is simple, which makes this one of the most common jobs in NLP development and one of the easiest places to compare methods fairly.

Before comparing methods, pin down the shape of the task, because it affects every later decision:

Single-label. Each text gets exactly one label, so the categories need to be mutually exclusive.
Multi-label. A text can have several labels at once, such as a review that mentions both delivery and price.
Hierarchical. Labels sit in a tree, such as billing and then refunds. Errors near the top of the tree usually matter more than errors near the leaves.
Open or changing. New categories appear regularly, which favours methods that do not need retraining for each one.

Write a short definition and a few examples for each label. If two people cannot agree on the correct label for a sample of real texts, no model will solve that. The categories themselves need fixing first.

The traditional options

"Traditional" covers a range of methods, and they are not all equally simple.

Rules and keyword patterns

Regular expressions and keyword lists work well when language is predictable, such as invoice numbers or fixed form wording. They are fast and easy to explain, and they make no attempt to guess. They become hard to maintain as exceptions accumulate, and they miss paraphrases.

Linear models on word features

TF-IDF features with logistic regression or a linear support vector machine make a strong baseline. They train quickly, run cheaply on ordinary hardware and let you inspect which words influenced a decision. They do not understand word order well, so they can struggle with negation or subtle phrasing, and they need labelled examples for every class.

# Illustrative only. A baseline using scikit-learn.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline

baseline = make_pipeline(
    TfidfVectorizer(ngram_range=(1, 2)),
    LogisticRegression(max_iter=1000),
)
baseline.fit(train_texts, train_labels)
predictions = baseline.predict(test_texts)

Embeddings with a small classifier

An embedding model turns each text into a vector that reflects its meaning. Training a simple classifier on those vectors often handles paraphrase better than word features, and it usually needs less labelled data than training a language model from scratch. It is a useful middle ground, although individual decisions become harder to explain.

Fine-tuned transformer models

Fine-tuning a transformer on your own labels can capture context well and gives you a model you control. The price is more labelled data, more training effort and more infrastructure, sometimes including a GPU. It tends to make sense for a stable, high-volume task where the labelling investment will be repaid.

What an LLM changes

With an LLM you describe the categories in a prompt, optionally add examples, and ask for a label. That shifts the work considerably:

Little or no training data to start. You can try an idea quickly, although you still need labelled data to measure it.
Categories can change in the prompt. Adding a label does not need a training run, but it does need a new evaluation.
Varied, messy language is often handled well, including indirect phrasing and mixed topics.
It can give a short explanation. This can help a reviewer, but an explanation may not reflect how the label was really chosen, so treat it as a convenience and not as proof.

The drawbacks are real too. Each call adds latency and usage cost, which can dominate at high volume. Behaviour can shift when a provider updates a model, so pin a version where the provider allows it and re-run your evaluation when it changes. Text may be sent to a third party unless the model is self-hosted, which raises privacy questions. Self-reported confidence scores from an LLM are not reliable probabilities, so they should not be used as thresholds without being tested.

Side-by-side comparison

These are general tendencies. Your own data can and does overturn them.

Factor	Rules	Linear or embedding model	Fine-tuned transformer	LLM with a prompt
Labelled data to start	None	Moderate	Most	None for training; a test set is still needed
Cost per prediction	Very low	Low	Low to moderate	Usually highest
Latency	Very low	Low	Low to moderate	Usually highest
Adding a category	New rules	New labels, retrain	New labels, retrain	Edit the prompt, re-test
Handling varied language	Weak	Fair to good	Good	Often good
Explaining a decision	Easy	Fairly easy for linear models	Hard	Plausible but unverified

Writing a classification prompt that behaves

A classification prompt is closer to a specification than a conversation. A few habits make it more dependable:

Define each label in a sentence, and state what does not belong in it. Most errors occur between neighbouring categories.
Include examples of hard cases, not only obvious ones, and keep them balanced so that the prompt does not lean towards one label.
Provide an "other" or "unclear" label so that the model has an honest way out when nothing fits.
Constrain the output using a structured output feature if your provider offers one, and validate the result in your own code regardless.
Treat the input as data. The text being classified may contain instructions, so see LLM guardrails and prompt injection defence if it comes from untrusted users.

# Illustrative only. Never trust a label without checking it.
ALLOWED = {"billing", "technical", "account", "other"}

def parse_label(raw_output):
    label = raw_output.strip().lower()
    return label if label in ALLOWED else "needs_review"

Comparing methods properly

The only reliable way to choose is to run the candidates on the same held-out test set, drawn from real data and labelled against your written definitions. Do not write prompts while looking at the test set, as this inflates the result in the same way that tuning a traditional model against it would.

Avoid relying on overall accuracy. If most tickets are about billing, a model that always answers "billing" can look strong while being useless. Report precision and recall for each class, a macro-averaged F1 score that weights rare classes equally, and a confusion matrix showing which labels get mixed up. Then weigh the outcomes by what mistakes actually cost: missing an urgent safety complaint is not the same as tagging a blog post slightly wrongly.

Include latency and running cost at your expected volume in the comparison, and read a sample of the errors. Sometimes the disagreement reveals mistakes in your labels and not in the model.

Combining methods in a cascade

The choice is rarely all or nothing. A cascade sends each text to the cheapest method able to handle it:

Rules settle the unambiguous cases, such as a fixed form subject line.
A trained model handles the common cases when its calibrated probability is high enough.
An LLM takes the remaining harder cases.
A person reviews what is still uncertain or high-risk.

LLMs can also help without being in the production path at all. They can draft labels for people to correct, which speeds up the creation of training data for a cheaper model. Those drafts can carry the LLM's biases, so keep a human-labelled test set that is independent of them.

A practical way to choose

Stable categories, plenty of labels, high volume: start with a linear baseline, then consider embeddings or fine-tuning if it falls short.
Few or no labels, or categories that change often: start with an LLM, and collect its reviewed output as future training data.
Fixed formats: try rules first.
Strict privacy or offline requirements: favour a model you can run yourself.

Whichever you choose, language and categories drift over time. Sample live predictions for review, and compare the distribution of labels over time so that a sudden shift gets noticed. If your task goes beyond assigning a label to pulling out specific values, read named entity recognition and data extraction with LLMs. If you are putting a classifier behind an application, adding an LLM feature to a full stack app covers the surrounding decisions.

Frequently asked questions

Is an LLM better than a traditional classifier?

Not automatically. An LLM is quick to try and often copes well with varied language, while a traditional model can be cheaper, faster and more predictable on a stable task with labelled data. Testing both on the same held-out data gives a practical answer.

Do I need labelled data if I use an LLM?

You may not need any for training, but you still need a labelled test set. Without one, you have no objective way to know how well the prompt works or whether a later change made it worse.

Can I trust the confidence score an LLM gives?

Not as a probability. Self-reported confidence is not necessarily calibrated, so test it against labelled data before using it as a threshold. A trained model's probabilities may need calibrating as well.

What metrics should I use for text classification?

Use precision and recall per class, macro-averaged F1 and a confusion matrix. Overall accuracy can look strong on imbalanced data even when rare but important classes are handled badly.

Text classification with LLMs vs traditional NLP