Prompt caching, and what it actually saves

Prompt caching is the largest single lever on most production LLM bills, and the one most often left switched off. It is also frequently oversold, so this covers both the arithmetic that makes it work and the workloads where it changes nothing.

How it works

When a provider processes your prompt, it builds an internal representation of it. If your next request begins with exactly the same prefix, that work can be reused instead of repeated. Caching exposes that reuse as a price: tokens served from cache bill at a fraction of the normal input rate.

The constraint is that it is a prefix cache. The match must start at the very beginning of the request and run contiguously. Change one character near the top and everything after it is a cache miss. This single fact determines how you should structure a prompt, and it is covered below.

The write premium and the read discount

Two rates, not one. Writing content into the cache costs slightly more than normal input — commonly 1.25× the base rate for a short-lived cache, or 2× for a longer-lived one. Reading from it costs dramatically less, typically 0.1× the base rate.

So a cached prefix is not free, it is roughly a tenth price after you have paid a small premium once. Every current figure for the models tracked here is in the model price table, in the cached-input column, with the provider URL each was read from.

When it pays for itself

Almost immediately. With a 1.25× write premium and a 0.1× read rate, you pay 0.25 extra units on the write and save 0.9 units on each read. One cache hit and you are ahead. With a 2× write premium it takes two hits.

The practical question is therefore not whether caching pays off but whether your cache entries survive long enough to be read at all. Cache lifetimes are short by default — often five minutes, refreshed on each hit. A high-traffic endpoint keeps its prefix warm continuously. An endpoint called twice an hour pays the write premium every time and never gets the discount, which makes caching a small net loss.

Model both cases with the cache slider in the cost calculator. For conversational products the effect is largest, because every turn after the first resends a prefix by construction — the chat cost estimator has a toggle for exactly this comparison.

Structuring a prompt to be cacheable

Since the match must start at the beginning and run contiguously, the rule follows directly: stable content first, variable content last.

• System prompt at the top. Identical on every request, so it is the ideal cache prefix.
• Tool definitions next. Also fixed by construction, and often larger than people realise — see the agent builder for what a tool schema actually weighs.
• Long reference documents after that, if they are the same across requests.
• User input last. Always.

The classic mistake is a timestamp, a request ID or a randomised greeting near the top of the system prompt. One variable token at position five invalidates the entire cache for every request, and the symptom is simply that caching appears not to work.

When it will not help

Four situations where caching is not the answer:

• Every request is unique. One-shot document processing with no shared prefix has nothing to cache. Look at the other eight levers instead.
• Low, irregular traffic. Entries expire before they are read.
• Short prompts. Below the provider's minimum cacheable length — typically around a thousand tokens — caching does not engage at all.
• Output-dominated workloads. Caching only touches input. If your bill is mostly generation, it barely registers, and constraining output length is the change that matters.

The thing caching does not fix

Cached tokens still occupy the context window. Caching changes what you pay for those tokens, not whether they count against the limit. A 100,000 token cached document is still 100,000 tokens of window unavailable for conversation history or retrieved passages — check the fit in the context window calculator.

It also does not make a bloated system prompt acceptable. At a tenth price a thousand wasted tokens are cheap rather than free, and they are still displacing something more useful. Price yours per paragraph in the system prompt analyzer.