How to reduce LLM API costs

Nine changes, ordered by what they return per hour of work rather than by how clever they are. Most teams find forty percent in the first three, and the last two are only worth reaching for at real scale.

Measure before you optimise

Almost every team that believes their bill is driven by request volume discovers it is driven by one endpoint sending an entire document where a paragraph would do. Before changing anything, get three numbers: median input tokens per request, median output tokens, and requests per month, broken down by endpoint.

Measure a real production payload in the token counter rather than estimating. Estimates in this domain are wrong in the expensive direction, because the rule of thumb everyone uses describes English prose and most payloads are not English prose.

1. Prompt caching

The highest-return change available on almost any repetitive workload. If your requests share a stable prefix — a system prompt, a document, a set of tool definitions — providers will bill that portion at roughly a tenth of the base input rate once it is cached.

A chatbot with a 2,000 token system prompt and 200 tokens of user text is around 90% cacheable. That is not a marginal improvement; it is most of the input line of the bill. Model it with the cache slider in the cost calculator, and read prompt caching explained for the break-even arithmetic and the cases where it does nothing.

2. Payload minification

Pretty-printed JSON spends two to four tokens per line on indentation alone. On a deeply nested payload this is thirty to fifty percent of the tokens, carrying no information the model uses.

Minify before sending; keep the readable version in your logs where a human reads it. The prompt optimizer does the transformation and shows the annual value at your volume. One caveat worth repeating: do not strip spaces between words. In most BPE vocabularies the leading space belongs to the word token, so removing it raises the count.

3. Trim the system prompt

A system prompt is paid for on every request by every user forever. A hundred tokens of boilerplate that changes nothing about the output is a permanent line item, and system prompts accumulate — someone hits a failure, adds an instruction, and never removes the one already covering it.

The system prompt analyzer prices each paragraph per year at your request volume. Sort by cost, not by how wrong each section feels; the expensive paragraph is rarely the one you suspected. Teams routinely find twenty to forty percent on a first pass through a prompt that several people have edited.

4. Route to smaller models

Most requests in most products are not hard. Classification, extraction, routing and formatting are handled well by models costing a twentieth of the flagship rate. Reserve the expensive model for the requests that genuinely need judgement.

The implementation is usually a cheap first pass with an escalation path when confidence is low. Compare rates in the model price table — the spread between the cheapest and dearest model tracking here is more than two orders of magnitude, and much of that gap is available to you without any quality loss on the easy half of your traffic.

5. Constrain output length

Output costs four to six times input. A model that answers in 800 tokens where 200 would do is the single most overlooked cost in most deployments, because nobody looks at output length as a metric.

Two fixes, both cheap: set max_tokens to something realistic rather than the model maximum, and ask for the format you want explicitly. "Reply with one sentence" and "reply with JSON matching this shape" both cut output dramatically compared to an open-ended request. Reasoning models deserve particular attention here, since their internal thinking is billed as output even though you never see it.

6. Cap conversation history

In a chat product, every turn resends the full history, so total input grows with the square of the turn count. Capping at the last N turns converts that back to linear growth, and summarising older turns into a compact state preserves most of the context at a fraction of the tokens.

Quantify it for your own conversation shape in the chat cost estimator. The gap between what a per-request estimate predicts and what you actually pay is commonly three or four times.

7. Retrieve less, better

In a retrieval-augmented system the retrieved passages usually dominate every other part of the prompt. Retrieving eight chunks of 400 tokens is 3,200 tokens on every request, billed every time.

Better chunking beats more retrieval on both cost and quality — see chunking strategies for RAG. Reranking and returning the top three rather than the top ten is often free accuracy alongside the saving, because irrelevant context degrades answers as well as costing money.

8. Batch what is not urgent

Most providers offer around 50% off for asynchronous batch processing with a latency budget measured in hours. Anything not blocking a user — nightly classification, backfills, evaluation runs, embedding generation — should be there.

This is pure margin with no quality trade-off, and it is skipped mostly because it requires a slightly different code path.

9. Lower agent iteration limits

An agent permitted ten iterations can make ten billed requests for one user action, each carrying the full tool-definition overhead plus accumulated history. Agents that cannot finish in four passes usually cannot finish in ten either; they just cost more to fail.

Price the configuration overhead in the agent builder — the tool schemas alone frequently exceed a thousand tokens per request — and see what an AI agent actually costs to run for the full breakdown.

What not to bother with

Two things get suggested constantly and are not worth your time. Shortening variable names or removing punctuation from prompts saves a rounding error and makes the prompt worse. And switching providers purely on headline input rate ignores output pricing, cached rates and the tokenizer difference — a model producing 30% more tokens for the same text at a 20% lower rate is more expensive, not less. Compare properly in the cost calculator.