Why chatbot costs grow faster than your user count

You modelled a chatbot at ten turns per conversation, priced one turn, multiplied by ten, and the real bill came in three times higher. Nothing was misconfigured. The model of the cost was simply the wrong shape.

The API has no memory

Language model APIs are stateless. The model retains nothing between requests. To hold a conversation, your client resends the entire history every single turn — the system prompt, every user message so far, and every assistant reply so far.

Turn ten therefore pays for turns one through nine again. Turn twenty pays for nineteen. Every token in the conversation is billed once for every subsequent turn it survives into.

The arithmetic of resending

Take a concrete case: an 800 token system prompt, 60 token user messages, 220 token replies, twelve turns.

The naive estimate is twelve requests of 860 input tokens each — 10,320 input tokens. The reality is that turn n carries the system prompt plus all 280 tokens from each of the previous turns, so the total billed input approaches 30,000. Roughly three times the estimate, and the multiple grows with every turn you add: total input scales with the square of the turn count, not linearly.

Output is the well-behaved half. It is generated once per turn and never resent, so it scales linearly. This tilts long conversations towards input-dominated economics, which has a practical consequence: a model with cheap input and expensive output can beat one with the reverse for a chat product, even when the opposite is true for one-shot work. Run your own numbers through the chat cost estimator, which shows the naive figure and the real one side by side.

The system prompt multiplier

The system prompt is the worst offender, because it is resent on every turn of every conversation. An 800 token system prompt in a twelve-turn conversation is 9,600 billed tokens — nearly a third of the total input, for text that never changes.

This inverts the usual intuition about where to spend effort. Trimming 300 tokens from a system prompt in a chat product is worth twelve times what it is worth in a one-shot product. The system prompt analyzer prices each paragraph annually so you can see which ones are earning their place.

Four fixes that work

1. Prompt caching

The resent prefix is exactly what caching is for, and a conversation produces one by construction. Typically a tenth of the base input rate. It does not change the quadratic shape — the growth is still quadratic, just with a much smaller coefficient. Note that caches expire, commonly after five minutes, so a user who steps away and returns pays full price on the next turn. Details in prompt caching explained.

2. Cap the history

Sending only the last N turns converts quadratic growth back to linear. The cost is that the model forgets earlier context — for many products invisible, for some unacceptable. Start at ten turns and see whether anyone notices.

3. Summarise instead of replaying

Compact older turns into a short state summary and send that instead of the transcript. Most agent frameworks now provide this as compaction. It preserves most of the useful context at a fraction of the tokens, and it is the standard answer for sessions that genuinely need long memory.

4. End conversations deliberately

A product encouraging a fresh session per task costs dramatically less than one where a single thread runs for weeks — and it often works better, because recall degrades as a window fills. This is a product decision rather than an engineering one, which is precisely why it tends not to get made.

The other wall

Cost is the first limit you hit. The context window is the second, and it arrives abruptly: a conversation that grows unbounded eventually exceeds it and the request fails outright. Work out where that happens for your turn length in the context window calculator.

Agent loops behave identically, for the same reason — each iteration resends the accumulated history, on top of a fixed tool-definition overhead that most frameworks never surface. See what an AI agent actually costs to run.