RAG
Retrieval-augmented generation (RAG)
RAG finds passages relevant to a question and places them in the prompt, so the model answers from supplied material rather than from training data.
Also written: RAG
The pipeline is: split documents into chunks, embed them, store the vectors, embed the query, retrieve the nearest chunks, and put them in the prompt.
Retrieved context usually dominates every other part of the prompt. Eight chunks of three hundred tokens is 2,400 tokens on every single request, billed every time.
In practice
When a RAG system gives disappointing answers, the cause is usually retrieval rather than generation — the correct chunk was never in the context, and no amount of prompt tuning can recover a fact that was not supplied. Measure the retrieval step separately, on questions with known correct sources, before touching the prompt.
Common questions
Is RAG better than fine-tuning?
For knowledge, yes. Retrieval supplies facts that change; fine-tuning teaches format and behaviour. If the correct answer would be different next month, it belongs in retrieval — a fine-tune bakes it in at training time.
Why does my RAG system give wrong answers?
Usually because the correct chunk was never retrieved, not because the model reasoned badly. Measure retrieval separately on questions with known correct sources before tuning the prompt.
Measure it
- Text Splitter for RAG and EmbeddingsToken-accurate chunking with configurable overlap. Respects sentence and paragraph boundaries.
- RAG Retrieval Budget CalculatorWhat retrieved context costs per month, and the k comparison table.
- RAG Context PackerScore-ordered packing into a fixed budget, with drop reasons reported.