Comparison
RAG vs fine-tuning
Retrieval adds knowledge; fine-tuning changes behaviour. Which to use, what each costs, and why most teams reach for the wrong one first.
The short answer
Use retrieval when the model needs facts it does not have, and fine-tuning when it needs to behave differently. Retrieval adds knowledge that can be updated in minutes; fine-tuning changes format, tone and consistency but bakes knowledge into weights you cannot edit. If your problem is that the model does not know something, fine-tuning is almost certainly the wrong tool.
At a glance
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model knows, per request | How the model behaves, permanently |
| Updating information | Re-index a document — minutes | Retrain the model — hours to days |
| Upfront cost | Embedding the corpus, usually tens of dollars | Training run plus dataset construction |
| Per-request cost | Higher — retrieved chunks are billed every time | Lower — the prompt gets shorter |
| Auditability | High — you can cite the passage used | Low — behaviour is in the weights |
| Iteration speed | Immediate | A training run per change |
| Handles new facts | Yes, immediately | No, not without retraining |
When to choose which
Choose RAG when
- The information changesDocumentation, prices, policies, inventory, anything with a date on it. A fine-tuned model is frozen at its training data and will confidently state last quarter’s policy forever.
- You need to cite sourcesRetrieval lets you show which passage an answer came from. That is often a product requirement and occasionally a legal one, and fine-tuning cannot provide it.
- The corpus is large or growingRetrieval scales by adding documents to an index. Fine-tuning on a large corpus is expensive and has to be redone as the corpus changes.
- You are still learning what the system needsIteration on retrieval is immediate. Iteration on fine-tuning is a training run each time, which turns a day of experiments into a fortnight.
Choose Fine-tuning when
- You need a consistent output format at volumeWhen the same structure must come back every time and prompt instructions are not holding, fine-tuning is more reliable than a longer prompt — and cheaper per request, because the instructions leave the prompt.
- The prompt has grown enormousIf you are shipping two thousand tokens of instructions and examples on every request, moving them into the weights removes that from every call. This is where fine-tuning pays back.
- You need a specific tone or styleVoice is hard to specify in instructions and easy to demonstrate in examples. A few hundred consistent examples teach it better than a page of description.
- A smaller model would do if it were specialisedFine-tuning a cheap model on a narrow task frequently matches a flagship on that task at a fraction of the price. This is the strongest economic case for it.
What it costs either way
Retrieval costs almost nothing to set up and something on every request. Embedding a corpus of fifty thousand documents runs to tens of dollars; the recurring cost is the retrieved chunks in every prompt, which typically dominate a RAG bill.
Fine-tuning costs something once and less on every request. The training run plus the dataset work is the investment; the return is a shorter prompt, offset by a higher per-token inference rate on the tuned model.
The break-even is therefore about prompt length and volume. If fine-tuning removes nine hundred tokens from a prompt sent two hundred thousand times a month, it pays back quickly. If it removes nothing, it never does.
The cost nobody budgets is the dataset. Building and labelling several hundred consistent examples is usually more expensive than the training run itself, and it has to be redone when the task changes.
The mistake people make
Fine-tuning to add knowledge
This is the most common and most expensive error in the decision. Fine-tuning teaches patterns, not facts — a model trained on your documentation learns to sound like your documentation, and will still invent plausible details. Worse, it now does so confidently and without a citation you can check. Knowledge belongs in retrieval, where it can be updated and traced.
How to decide
- 1Ask what is failing. If the model does not know something, that is retrieval. If it knows but answers in the wrong shape, that is fine-tuning.
- 2Try the cheaper options first: a better prompt, then few-shot examples, then retrieval. Each takes hours rather than weeks.
- 3If the prompt is now large and stable and the format still drifts, price fine-tuning against the tokens it would remove.
- 4Consider doing both. A fine-tuned model that handles format, fed by retrieval that supplies facts, is a common and sensible end state.
Price it yourself
- Fine-Tuning Cost and Payback EstimatorTraining cost against the prompt tokens it removes, with payback in months.
- RAG Retrieval Budget CalculatorWhat retrieved context costs per month, and the k comparison table.
- Text Splitter for RAG and EmbeddingsToken-accurate chunking with configurable overlap. Respects sentence and paragraph boundaries.
Frequently asked questions
- Can I use RAG and fine-tuning together?
- Yes, and it is often the right answer. Fine-tune for format and tone so the prompt can be short, and retrieve for facts so they stay current. The two solve different problems and do not conflict.
- Which is cheaper?
- Retrieval is cheaper to start and more expensive per request. Fine-tuning is the reverse. At low volume retrieval almost always wins; at high volume with a large stable prompt, fine-tuning can overtake it.
- How many examples do I need to fine-tune?
- Useful results generally start in the high hundreds for a narrow task. Consistency matters far more than volume — a hundred examples that agree with each other beat a thousand that do not.