Checklist · Shipping
Model Migration Checklist
Migrations fail in the details. The prompt transfers, the API shape mostly transfers, and then tool calling behaves differently, the token count moves fifteen percent, and a refusal arrives where none used to.
Run this before you route traffic, not after.
Ticks are saved in this browser only — close the tab and come back to where you were. Nothing is sent anywhere and there is no account.
0 of 19 done · 6 critical outstanding
Measure first
0/4Without a baseline, "the new model seems fine" is the entire evidence for the migration. It usually is fine on the cases you remember.
Check this with the eval dataset builderComparing published prices compares a price per million tokens, not a price per request. Different tokenizers, different prompt formats and different verbosity all move the second number.
Check this with the llm cost calculatorA cheaper model that is slower can cost more in abandoned sessions than it saves per token.
Tokenizers differ by ten to twenty percent on the same text, and more on code or non-English. Price per million is only comparable after that adjustment.
Check this with the encoding comparator
Compatibility
0/6Context windows are shared between input and output on most APIs. A prompt that fits with no room to answer does not fit.
Check this with the context window calculatorProviders differ on whether the system prompt is a message, a parameter, or absent. A system prompt sent as a user message is not a system prompt.
This is where most migrations break. The schema shape, the result message shape and the parallel-call behaviour all differ between providers.
Check this with the openai anthropic converterThe names differ and so does the set. Code that checks for one provider's string silently stops detecting truncation on another.
Schema enforcement, JSON mode and prompt-only requests have very different reliability, and the fallback is silent.
Check this with the tool schema builderTemperature is not calibrated across model families. The same value produces different variance, and a value near the top of one range may be out of range in another.
Behaviour differences
0/4Models differ in how much they wrap answers in commentary, headings and markdown. Downstream parsers notice immediately.
Safety training differs. Content that passed for a year can start being declined, and the failure looks like a bug rather than a policy.
A model that answers thirty percent longer at the same price per token is thirty percent more expensive on output, whatever the price sheet says.
Reasoning tokens can dominate both cost and time to answer, and they do not appear in the visible output where you would notice them.
Check this with the reasoning token estimator
Rollout
0/5Evaluation sets contain the cases you thought of. Shadow traffic contains the ones you did not, at no risk.
A regression visible at five percent of traffic is a rollback. The same regression at a hundred percent is an incident.
The value of a rollback plan is inversely proportional to how long it takes to execute.
Migrating to a floating alias means doing this again without being told.
Monthly and quarterly workloads only appear once. Decommission after you have seen one, not before.
Tools for this list
- o200k_base vs cl100k_base ComparatorBoth encodings, both exact. Matters on any model migration.
- LLM Model FinderSet a budget and a context requirement, get the shortlist. Sorted by your own input-to-output ratio.
- OpenAI to Anthropic Format ConverterPort a payload between providers. Handles the system prompt difference.
- Context Window CalculatorPaste your context. See which models swallow it whole and which will reject the request.
Questions
Can I reuse my prompts unchanged?
Usually as a starting point, rarely as a finished one. Prompts get tuned to a model's quirks over time, and some of that tuning becomes noise or harm on a different model. Re-run your eval set before assuming.
How long should a migration take?
For a single well-evaluated feature, one to two weeks including shadow traffic. Without an eval set, budget longer — most of the time goes into building the thing that tells you whether it worked.
Is it worth migrating for a small price difference?
Rarely below about thirty percent. Below that the engineering time, the risk and the re-tuning usually cost more than the saving in the first year.