RAG vs fine-tuning: which to choose

The question to ask first

RAG and fine-tuning are often presented as rivals, yet they change different things. RAG alters the input: for each question, the system finds relevant text and adds it to the prompt. Fine-tuning alters the model: it continues training from an existing model using your examples, so that its weights shift towards the behaviour those examples show.

So the useful first question is not which technique is better. It is what kind of failure you are looking at. Collect some poor outputs and sort them into two groups:

• A knowledge failure. The answer is wrong or vague because the model never saw the facts. It cannot state your returns policy, or it describes last year's pricing rules.
• A behaviour failure. The facts were available, but the output is in the wrong shape. It uses the wrong labels, ignores your house style or handles a repeated judgement inconsistently.

A simple test separates the two. Paste the relevant facts directly into the prompt by hand and run the request again. If the answer becomes good, the model needed knowledge, and retrieval could supply it automatically. If the answer is still poor with all the facts in front of the model, retrieval alone will not fix it, and you should look at the prompt, the examples or the model itself.

How the two compare

Provider support and pricing for fine-tuning vary by model and change over time, so check them for the specific model you plan to use. Some hosted models do not offer it at all, which can settle the question on its own.

When RAG is the better choice

RAG suits problems in which the answer lives in content you hold. Look for these signs:

• The information changes. Policies, documentation and product details are revised, and an index can reflect an edit without any training.
• People need to verify answers. Because the system knows which passages it supplied, it can show them alongside the answer.
• The content is large. Retrieval selects a small relevant portion for each question rather than sending everything.
• Access differs by user. A filter applied outside the model can restrict what each person's question is able to retrieve.
• Removal matters. Deleting a document from an index is direct. Removing a fact from a trained model is not.

The price is a pipeline that has to be built and looked after. Retrieval quality depends on parsing, on how documents are chunked and on the search method, and retrieved text is untrusted input that may contain instructions. The guide on what RAG is and when to use it covers these parts, as well as the cases where it is the wrong tool.

When fine-tuning is the better choice

Fine-tuning is worth considering when you can describe the desired behaviour mainly through examples, and when instructions have stopped improving it. Reasonable candidates include:

• A narrow, repeated task. Classification or extraction with stable labels gives the model one pattern to learn. The guide to text classification with LLMs and traditional NLP compares this with other methods.
• A strict output style. A format or tone that is hard to specify in words may be easier to show.
• Long prompts carrying many examples. Moving repeated examples into training may shorten each request. Whether that saves money depends on the rates and traffic involved, so work it out for your case.
• A move to a smaller model. A tuned smaller model might handle a narrow task acceptably at lower cost or latency, although this has to be demonstrated on your own test set.

The price is a different kind of maintenance. You need enough good examples, a way to keep test data separate from training data, and a process for repeating the work when requirements change or the base model is retired. Errors in your examples are also learned. Inconsistent labels, in particular, are often a sign that the task definition needs work before any training begins.

Why fine-tuning is a poor way to add facts

It is tempting to think of fine-tuning as a way of teaching a model your documents. In practice, this tends to disappoint for several reasons. Training adjusts the model towards patterns, and it does not store documents for reliable lookup, so the model can blend what it learned into a confident and incorrect answer. A tuned model also cannot tell you where a statement came from. Each change in the facts requires new training, and in the meantime the old facts remain in the weights.

There is a privacy concern as well. Confidential text in a training set may be reproduced in later output, and every user of that model has potential access to it. If this matters, retrieval with enforced permissions gives you a clearer boundary, though it still needs defences of its own.

Try the cheaper fixes first

Before you commit to either technique, check whether a simpler change would solve the problem:

1. Improve the prompt. Clearer instructions and a few well-chosen examples resolve many behaviour failures.
2. Use structured output. Where a provider or library can enforce a schema, a format problem may need no training at all.
3. Add static context. If the knowledge is small and stable, including it in the prompt avoids a retrieval system. Prompt caching, where available, may reduce the cost of repeating it.
4. Use a tool call. Live values and calculations are usually better fetched from an API or database than retrieved from an index or learned in training.
5. Try another model. Compare a different base model on your test set, because this may be less work than tuning the current one.

Using both together

The two can work together because they address different layers. A support assistant might use retrieval for current policy, plus a tuned model for the structure and tone of its replies. The tuned model supplies the behaviour, while the index supplies the facts.

If you combine them, keep the training examples realistic. Include retrieved passages in the examples, including some cases in which the passages do not answer the question and the correct output is to say so. Otherwise, the model may never learn to decline, and the training data will not resemble what it sees in production.

Add the second technique only after the first has reached its limit, because each adds its own source of failure. If you change both at once, you will struggle to work out which one caused a result to move.

A way to decide on your own data

A short experiment gives a better answer than general advice:

1. Build a test set. Gather realistic requests with expected outcomes, and keep them apart from anything used for training.
2. Record a baseline. Run your current prompt and model, and label each failure as knowledge or behaviour.
3. Run the paste test. Supply the correct facts by hand for the knowledge failures, and confirm that the answers improve.
4. Change one thing. Add retrieval or a tuned model, not both, and score it on the same test set.
5. Compare the running cost. Look beyond quality to latency, price and the effort of keeping the system current.

For retrieval, score retrieval and answers separately, as described in how to evaluate a RAG system. Where the failures are mostly knowledge failures, the next step is usually to prototype a small pipeline, and the guide to building a RAG chatbot in Python walks through one.

If your diagnosis points towards retrieval and you want help planning it, the RAG development page describes what I can help with.

Frequently asked questions

Is RAG cheaper than fine-tuning?

It depends on your volume, provider and content. RAG has indexing, search and extra prompt costs on each request, while fine-tuning has dataset and training costs and may change the price per request. Estimate both using current prices.

Can fine-tuning replace RAG?

Generally not when answers depend on specific or changing facts. A tuned model cannot cite a source or reflect an update without further training, and it cannot apply per-user access rules to what it learned.

Can I use RAG and fine-tuning together?

Yes. Retrieval can supply the facts while the tuned model handles format, labels or tone. Make the training examples include retrieved passages so that they resemble real requests, and add one technique at a time.

How do I know which one my problem needs?

Paste the correct facts into the prompt by hand and try again. If the output becomes good, the model needed knowledge, which points to retrieval. If it stays poor, look at the prompt, examples or model.