Tokens
Byte-pair encoding (BPE)
Byte-pair encoding builds a tokenizer vocabulary by repeatedly merging the most frequent adjacent pair of symbols in a corpus until it reaches a target size.
Also written: BPE
The procedure is simple enough to state in a sentence: start with every individual byte as its own token, find the most frequent adjacent pair in the training corpus, merge it into a new single token, and repeat.
The consequence is that efficiency tracks frequency in the training data. The word "the" earned a merge early and is one token. A UUID never appeared twice and is a dozen. This is why English prose is cheap and identifiers, indentation and non-Latin scripts are not.
In practice
BPE is why " the" with a leading space is one token and "the" at the start of a line is another. It is also why an unusual product name — "Zylthorix" — costs four or five tokens while "engineering" costs one: frequent sequences earned a merge during training, rare ones are rebuilt from fragments every time you send them.
Common questions
Why does BPE split some words and not others?
Frequency in the training corpus. Sequences that appeared often enough earned a merge into a single token; everything else is rebuilt from smaller pieces each time. It is a learned artefact, not a rule you can derive.
Does BPE handle languages it was not trained on?
It never fails, but it gets expensive. Text outside the training distribution falls back to byte-level fragments, so the same meaning can cost two or three times more tokens than its English equivalent.