TokenPad

RAG

Embedding dimensions

The dimension count is the length of an embedding vector — commonly 384, 768, 1536 or 3072 — and it drives storage and search cost directly.

At 1,536 dimensions in float32 each vector is about six kilobytes. Half a million chunks is three gigabytes of raw vectors before metadata or index overhead.

Many current models support reducing dimensions at negligible accuracy cost, and the smaller vector is both cheaper to store and faster to search.

In practice

Dimension is a direct multiplier on storage and on query latency. Ten million chunks at 1,536 dimensions is roughly 61 GB of raw vectors; at 768 it is about 30 GB. Several current models are trained so the vector can be shortened with modest quality loss, which makes this a tuning decision rather than a fixed property.

Common questions

Can I reduce embedding dimensions?

Several current models are trained so the vector can be shortened with modest quality loss. Test retrieval quality at the shorter length on your own corpus before committing — the loss is corpus-dependent.

Do more dimensions mean better retrieval?

Up to a point, then storage and query latency grow faster than accuracy. At tens of millions of chunks the difference between 768 and 1,536 is a line item, so it is a trade rather than a maximisation.

Measure it

  • EmbeddingAn embedding is a vector of numbers representing the meaning of a piece of text, so that similar meanings sit near each other in that space.
  • Vector databaseA vector database stores embeddings and finds the nearest ones to a query vector, usually with an approximate nearest-neighbour index.
  • QuantizationQuantization stores model weights at lower numerical precision — 8-bit or 4-bit instead of 16-bit — to reduce memory and increase speed.

More in RAG