Why chat costs more than you budgeted
Language model APIs are stateless. The model has no memory of your last request, so to hold a conversation your client resends the entire history every single turn. Turn ten pays for turns one through nine again. Turn twenty pays for nineteen.
The consequence is that input tokens billed across a conversation grow with the square of the turn count, not linearly. A per-request calculator multiplies one turn by the number of turns and produces a number that can be three or four times too low. The comparison at the top of this page shows the gap for your own numbers.
The arithmetic
Take a 800 token system prompt, 60 token user messages and 220 token replies over twelve turns. The naive estimate is twelve requests of 860 input tokens: 10,320 tokens. The reality is that each turn also carries every previous message, so the total billed input approaches 30,000 tokens — roughly three times the estimate, and the multiple grows with every turn you add.
Output is the well-behaved half. It is generated once per turn and never resent, so it scales linearly. This is why long conversations shift the cost balance towards input in a way that single-request workloads never do, and why a model with cheap input can beat one with cheap output for conversational products even when the reverse is true elsewhere.
What prompt caching fixes, and what it does not
Every turn after the first resends a prefix the provider has already processed. That prefix is exactly what prompt caching is for, and it typically prices at around a tenth of the base input rate. Toggle it above and watch the total move.
What caching does not do is change the shape of the growth. The quadratic term is still there; caching lowers its coefficient. At twenty turns you are still resending twenty times more than you think, just at a discount. Caches also expire — commonly after five minutes unless you pay for a longer window — so a user who steps away and returns pays full price on the next turn.
The levers that actually work
- Cap the history. Sending only the last N turns converts quadratic growth back to linear. The cost is that the model forgets earlier context, which for many products is acceptable and for some is invisible.
- Summarise rather than replay. Compact older turns into a short state summary. Most agent frameworks now call this compaction; it is the standard answer for long-running sessions.
- Shorten the system prompt. It is resent on every turn of every conversation, so a 300 token reduction is multiplied by your entire traffic. The system prompt analyzer prices each paragraph of it individually.
- End conversations deliberately. A product that encourages a fresh session per task costs dramatically less than one where a single thread runs for weeks — and often works better, because recall degrades as the window fills.
The other limit
Cost is the first wall you hit; the context window is the second. A conversation that grows unbounded eventually exceeds it, and the failure is abrupt. Check where that happens for your turn length in the context window calculator, and if your product also does one-shot work, price that separately in the cost calculator — the two workloads have genuinely different economics and averaging them hides both.