Tokens
o200k_base
o200k_base is the byte-pair encoding used by GPT-4o and later OpenAI models, with roughly twice the vocabulary of the cl100k_base it replaced.
The larger vocabulary produces fewer tokens for the same text. The gain is modest on English prose and substantial on code and non-English content, which was a deliberate design goal rather than a side effect.
Its predecessor, cl100k_base, is still used by GPT-4 and GPT-3.5. The difference between the two on identical text is commonly ten to twenty percent.
In practice
o200k_base carries roughly 200,000 entries against cl100k_base's 100,000, and the practical effect is that the same English paragraph lands about 5–10% shorter on the newer encoding. The merge ranks themselves are around a megabyte compressed, which is why this site loads them only when you actually count something rather than on every page.
Common questions
Which models use o200k_base?
The newer OpenAI generation. Older GPT-4 era models use cl100k_base. Sending text through the wrong one produces counts that are close enough to look right and wrong enough to matter near a context limit.
Is o200k_base cheaper than cl100k_base?
It produces fewer tokens for the same English text — typically 5–10% fewer — because the vocabulary is twice the size. Whether that is cheaper depends on the price of the model using it, not on the encoding.