TokenPad

Recipe · Tokens

Count tokens in Python

Count OpenAI tokens exactly in Python with tiktoken, including the per-message overhead that makes a naive count too low.

The problem

You need to know how many tokens a prompt will cost before you send it — to stay inside a context window, to estimate spend, or to decide whether a payload needs trimming.

Counting the string is not enough. A chat request wraps every message in structural tokens, so a count of the content alone is consistently too low.

Exact count for a single stringpython
import tiktoken

# o200k_base for GPT-4o and later; cl100k_base for GPT-4 and GPT-3.5.
encoding = tiktoken.get_encoding("o200k_base")

def count_tokens(text: str) -> int:
    return len(encoding.encode(text))

print(count_tokens("Summarise this ticket in two sentences."))
Count a whole messages array, including overheadpython
import tiktoken

encoding = tiktoken.get_encoding("o200k_base")

# Each message carries a few structural tokens on top of its content,
# plus a couple for the reply priming. Ignoring these undercounts by
# roughly 4 tokens per message — which matters when you are near a limit.
TOKENS_PER_MESSAGE = 3
TOKENS_PER_REPLY = 3

def count_messages(messages: list[dict]) -> int:
    total = TOKENS_PER_REPLY
    for message in messages:
        total += TOKENS_PER_MESSAGE
        for key, value in message.items():
            if isinstance(value, str):
                total += len(encoding.encode(value))
    return total

messages = [
    {"role": "system", "content": "You are a support triage assistant."},
    {"role": "user", "content": "My card was charged twice."},
]

print(count_messages(messages))
Trim a conversation to fit a budgetpython
def trim_to_budget(messages: list[dict], budget: int) -> list[dict]:
    """Keep the system message and as many recent turns as fit."""
    system = [m for m in messages if m["role"] == "system"]
    rest = [m for m in messages if m["role"] != "system"]

    kept: list[dict] = []
    used = count_messages(system)

    # Walk backwards: recent turns matter more than old ones.
    for message in reversed(rest):
        cost = count_messages([message]) - TOKENS_PER_REPLY
        if used + cost > budget:
            break
        kept.insert(0, message)
        used += cost

    return system + kept

Why it is written this way

Pick the encoding, not the model name

tiktoken.encoding_for_model() works but fails on models it has not heard of, which includes anything released after your installed version. Naming the encoding directly is more robust: o200k_base for GPT-4o and later, cl100k_base for GPT-4 and GPT-3.5.

Load the encoding once

get_encoding downloads and parses the merge ranks. Doing it inside a function that runs per request is a common and expensive mistake — hoist it to module level.

This is exact for OpenAI only

Anthropic and Google publish no equivalent library that runs locally with the same fidelity. For those, either call the provider’s token counting endpoint or treat any local figure as an estimate.

What breaks the naive version

  • Counting only the user message. Your bill covers the system prompt, the whole conversation history, tool definitions and the response — in an agent loop the user message is frequently a small minority of the request.
  • Forgetting that max_tokens comes out of the same context window. Input that fits does not mean the request fits.
  • Assuming the count transfers to another provider. Each one trains its own vocabulary, and the difference is routinely 20–30%.

Check your numbers