What is a token in an LLM?

Every price, every rate limit and every context window in the language model industry is denominated in tokens. It is the only unit that matters commercially, and it is the one nobody can see in their editor. This is what one actually is.

What a token actually is

A token is an entry in a fixed vocabulary that the model was trained with. Before any text reaches the model it is converted into a sequence of integers, each one an index into that vocabulary. The model never sees characters or words. It sees numbers.

The vocabulary is typically 100,000 to 200,000 entries. Some are whole common words. Some are word fragments. Some are single characters, and a few are raw bytes for text the vocabulary has no better representation for. Crucially, the vocabulary is fixed at training time and never changes afterwards — which is why the token count for a given string is completely deterministic, and why you can compute it exactly without calling any API.

How byte-pair encoding builds them

Most current tokenizers use byte-pair encoding. The training procedure is simple enough to describe in a sentence: start with every individual byte as its own token, find the most frequent adjacent pair in the corpus, merge it into a new single token, and repeat until the vocabulary reaches the target size.

The consequence is that frequency in the training corpus determines efficiency. The word the appeared constantly, so it earned a merge early and is one token. A UUID never appeared twice, so it is rebuilt from a dozen fragments. This is not a rule you can reason about from first principles — it is a frozen record of what one particular corpus contained. Watching it happen is more instructive than reading about it: the tokenizer playground renders each token separately so you can see exactly where the splits land.

Why a token is not a word

For ordinary English prose, one token averages about 0.75 words, or roughly four characters. That ratio is the source of the ubiquitous "characters ÷ 4" rule of thumb, and it is fine as a sanity check on English prose.

It falls apart everywhere else:

• Code runs about 2.7 characters per token. camelCase identifiers split at the case boundary, and every operator and bracket takes its own token.
• Pretty-printed JSON runs about 2.2, because indentation consumes a token per level per line while carrying no information.
• Non-Latin scripts can fall below 1.6, sometimes costing two or three tokens per character.

The awkward part is that the payloads dominating real API bills — structured data, code, multilingual content — are precisely the ones the rule of thumb gets most wrong. If you want an estimate that respects the content type, the words to tokens converter shows all of them side by side.

Why models disagree on the same text

Each provider trains its own vocabulary, so the same sentence produces different counts on different models. Even within one provider the vocabulary changes between generations: OpenAI moved from cl100k_base to o200k_base with GPT-4o, roughly doubling the vocabulary and reducing token counts by ten to twenty percent for the same text.

Movement runs the other way too. Anthropic states that the tokenizer introduced with Claude 4.7 produces approximately 30% more tokens for the same input than its predecessor. A model can therefore become more expensive per page of text at an unchanged headline price — which is not a trick, just a consequence of the price being quoted in a unit that is itself model-dependent.

This has a practical implication worth stating plainly: a cost comparison between providers that uses one token count for both is wrong. The token counter counts the same text against every model at once for exactly this reason, and marks which counts are exact and which are estimates, because not every provider publishes a tokenizer that can run in a browser.

Where it costs you money

Three places, in descending order of how much they surprise people.

The bill

Input and output tokens are priced separately, and output typically costs four to six times more because it is generated one token at a time rather than processed in a single parallel pass. A model that answers concisely can be cheaper in production than one with half its headline rate.

The context window

The window is a single budget shared by the system prompt, the entire conversation history, tool definitions, retrieved documents and the response being generated. Exceeding it is a hard error, not a graceful truncation.

The compounding

Anything resent on every request multiplies. A system prompt is paid for on every call by every user forever; conversation history is resent on every turn, so chat costs grow with the square of the turn count. Both are covered in why chatbot costs grow faster than your user count.

The habit worth building is small: before shipping anything that runs at volume, count it. It takes ten seconds and it is exact.