Checklist · Shipping
RAG System Launch Checklist
Retrieval demos well and fails quietly. The demo asks questions whose answers are in the first chunk; production asks questions whose answers are split across two documents, or absent entirely.
Most of this list is about that second case.
Ticks are saved in this browser only — close the tab and come back to where you were. Nothing is sent anywhere and there is no account.
0 of 18 done · 6 critical outstanding
Chunking
0/5A chunk that starts mid-sentence embeds badly, because the embedding represents a fragment nobody would have written. It also reads badly when it is shown back to the model as context.
Check this with the text splitterOverlap rescues facts that straddle a boundary and duplicates everything else. Ten to fifteen percent is a common landing point; zero and fifty are both usually wrong.
Check this with the chunk overlap visualizerThe right size depends on your documents. Dense reference material wants small chunks; narrative wants large ones. The default in the first tutorial you read was chosen for neither.
Check this with the text splitterWithout them the model cannot cite, you cannot debug a bad answer back to a bad chunk, and the user cannot verify anything.
Half a table is worse than no table: it looks like data and is missing the rows that would have contradicted the answer.
Retrieval quality
0/5If the right chunk is not retrieved, no prompt improvement can fix the answer. Measure the retrieval step separately or you will spend weeks tuning the wrong stage.
Embeddings are weak exactly where users are precise: product codes, error numbers, names, version strings. Those are the queries where a plain keyword match wins outright.
More context is not more accuracy. Past a point the relevant chunk competes with nine irrelevant ones, and the irrelevant ones cost money on every request.
Check this with the context rot visualizerThis is where a retrieval system hallucinates. A model given no useful context and no permission to say so will produce something confident.
A retrieval index is a cache of your documentation. Every cache has an invalidation story, and a system that quotes last quarter's policy is worse than one that says it does not know.
Grounding
0/4Models will produce citation-shaped text for sources they did not use. The check is cheap and it is the difference between a citation and a decoration.
Retrieved documents are untrusted input. Anything indexed — including a page a user uploaded — can carry instructions.
Check this with the prompt injection scannerLeft unspecified, it will mix the two, and you will not be able to tell which sentence came from where.
This is the failure mode that survives every other check: correct-sounding, well-cited, and not actually supported by the source.
Cost and operations
0/4Indexing is a one-time number people remember and a recurring one they forget. Documents change, and every change is a re-embed.
Check this with the embedding cost calculatorRetrieval multiplies input length. A 30 token question with five 800 token chunks is a 4,000 token request, and that is the number you are billed on.
Check this with the llm cost calculatorDuplicated content wins retrieval by occupying several of the top slots with the same text, crowding out the chunk that had the rest of the answer.
Check this with the dataset deduplicatorSeveral current models support shortening the vector with modest quality loss. At tens of millions of chunks that choice is a material part of the infrastructure bill.
Check this with the vector storage estimator
Tools for this list
- Text Splitter for RAG and EmbeddingsToken-accurate chunking with configurable overlap. Respects sentence and paragraph boundaries.
- Chunk Overlap VisualizerShows the repeated text between chunks and the storage it costs.
- Embedding Cost CalculatorIndex build, re-indexing and query embedding, priced separately.
- Vector Database Storage EstimatorStorage by dimensions and precision, with quantisation savings shown.
Questions
What is the most common reason a RAG system underperforms?
Retrieval, not generation. In most systems that produce disappointing answers the correct chunk was never in the context, and every hour spent on the prompt was spent on the wrong stage.
Should I fine-tune instead?
Not for knowledge that changes. Fine-tuning teaches format and behaviour; retrieval supplies facts. If the answer would be different next month, it belongs in retrieval.
How many test questions do I need?
Thirty with known correct sources is enough to see whether retrieval works. A hundred is enough to see whether a change to chunking helped.