Models
Inference
Inference is running a trained model to produce output, as opposed to training, which is the process that produced the weights.
Every API call you make is inference. It happens in two phases: prefill, where the whole input is processed in parallel, and decode, where output tokens are generated one at a time.
That split explains the pricing and the latency. Prefill is fast and cheap per token, so it drives time to first token; decode is slow and expensive, so it drives both total latency and most of the bill on generation-heavy work.
In practice
Inference splits into two phases with different economics: prefill processes your input in parallel and is fast per token, generation produces output one token at a time and is not. That asymmetry is why a 20,000-token prompt with a 100-token answer feels quicker than a 2,000-token prompt with a 1,000-token answer.
Common questions
Why is the first token slower than the rest?
Prefill processes the entire input before generation starts. A long prompt means a long prefill, which is why time to first token scales with input length while the tokens after it arrive at a steady rate.
What makes inference cheaper?
Sending less, caching the stable prefix, capping output, and routing easy work to a smaller model. In that order, roughly, by saving per hour of effort.