What you are looking at
Each coloured block is one token — the smallest unit the model reads. Byte-pair encoding starts from individual bytes and repeatedly merges the most frequent adjacent pair until it has a vocabulary of a fixed size, around 200,000 entries for o200k_base. Sequences that were common in the training corpus earn a single token; anything unusual gets rebuilt from fragments.
That is why the is one token and a UUID is a dozen. The vocabulary is not a rule you can reason about from first principles — it is a frozen artefact of what the encoder happened to see. Looking at the split is the only reliable way to understand why a given payload is expensive.
Four patterns worth internalising
Leading spaces belong to the word
In most BPE vocabularies the token is " the", with the space attached, not "the". This is why concatenating strings without spaces, or stripping whitespace to "save tokens", usually makes the count worse: you force the encoder off its common merges and onto fragments.
Indentation is pure waste
Load the pretty-printed JSON sample and count the blocks spent on nothing but leading spaces. Minifying JSON before it enters a prompt routinely removes 20–30% of the tokens and loses no information the model can use. If you are sending structured data at volume, this is the cheapest optimisation available to you.
Code fragments in predictable places
camelCase and snake_case identifiers split at the case or underscore boundary; operators and punctuation each take their own token. Source code typically runs 2.5–3.0 characters per token against roughly 4.0 for prose, so a codebase costs meaningfully more per kilobyte than the same weight of documentation.
Non-English text pays a structural surcharge
The training corpora are English-dominated, so English words get dedicated tokens while other languages fall back to sub-word fragments or raw bytes. Compare the English and Spanish samples: near-identical sentences, materially different counts. For languages in non-Latin scripts the multiple is larger still, often two to four times. If your product serves several markets on one price, this is a real margin difference between them.
o200k_base and cl100k_base
cl100k_base is the older encoding, used by GPT-4 and GPT-3.5. o200k_base is the current one, with roughly twice the vocabulary, used from GPT-4o onwards. The larger vocabulary produces fewer tokens for the same text — noticeably so for non-English content, which was a deliberate goal.
Switch between the two on the same input and you will see counts differ by 10–20%. That difference is real money on a migration, and it is why a cost projection built on one encoding does not transfer to a model that uses another. Anthropic reports a comparable break in its own line: the tokenizer introduced with Claude 4.7 produces around 30% more tokens than its predecessor for identical text.
Turning this into a saving
The workflow that pays: paste a real production payload here, find the structural waste, remove it, then measure the trimmed version in the token counter and multiply the difference by your monthly volume in the cost calculator. A 25% reduction on the input side of a serious deployment is usually a four-figure annual saving, and it takes about ten minutes to find.