Cost
Model routing
Model routing sends each request to the cheapest model that can handle it, escalating to a more capable one only when needed.
Most requests in most products are not hard. Classification, extraction, routing and formatting are handled well by models costing a twentieth of the flagship rate.
Escalated requests are paid twice, once on each model. At typical price gaps of twenty to one and escalation rates in single digits, it is still a large net win.
In practice
Most production traffic is easy, and the price difference between tiers is often more than tenfold. Sending classification, extraction and routing to a small model while reserving the frontier tier for open-ended reasoning is typically the largest structural saving available — bigger than caching in absolute terms, though it takes a week rather than an hour.
Common questions
How do I decide which requests go to a cheaper model?
Start with task type rather than a classifier: classification, extraction, routing and formatting to the small model, open-ended reasoning to the large one. Add a confidence-based escalation path afterwards if needed.
How much does model routing save?
It depends on the traffic mix, but the price gap between tiers is often more than tenfold and most production traffic is easy. In absolute terms it is usually the largest structural saving available.
Measure it
- Model Routing Savings CalculatorTwo-tier routing with escalation cost, and the break-even escalation rate.
- LLM Model FinderSet a budget and a context requirement, get the shortlist. Sorted by your own input-to-output ratio.
- LLM API Cost CalculatorRequests per month in, dollars out. Input, cached input and output priced separately.