TokenPad

RAG

Vector database

A vector database stores embeddings and finds the nearest ones to a query vector, usually with an approximate nearest-neighbour index.

Storage is dimensions times bytes per value, times the number of chunks, plus metadata and roughly thirty percent for the index structure. At 1,536 dimensions in float32 that is about six kilobytes per chunk before anything else.

Quantisation is the largest lever: float16 halves storage and int8 quarters it, usually at a small recall cost worth measuring on your own data.

In practice

The dimension choice is an infrastructure decision, not a detail. Ten million chunks at 1,536 dimensions in float32 is roughly 61 GB before indexes; at 768 it is half that. Several current embedding models support shortening the vector with modest quality loss, and at that scale the difference shows up in the hosting bill every month.

Common questions

Do I need a vector database?

Below roughly a hundred thousand chunks, usually not — an array and a brute-force similarity scan is fast enough and far simpler to operate. The database earns its complexity at scale, not at the start.

How much storage do embeddings need?

Dimensions times four bytes per vector in float32, before indexes. Ten million chunks at 1,536 dimensions is about 61 GB; at 768 it is roughly half, which is why dimension is an infrastructure decision.

Measure it

  • EmbeddingAn embedding is a vector of numbers representing the meaning of a piece of text, so that similar meanings sit near each other in that space.
  • Retrieval-augmented generation (RAG)RAG finds passages relevant to a question and places them in the prompt, so the model answers from supplied material rather than from training data.
  • RerankingReranking scores a wide set of retrieved candidates with a more precise model and keeps only the best few for the prompt.

More in RAG