Cost
Time to first token (TTFT)
Time to first token is the delay between sending a request and the first token arriving — dominated by the time spent processing your input.
Also written: TTFT
Before generating anything the model must prefill: process your entire prompt in one pass. On a long prompt that prefill is usually the largest component of the wait.
With streaming, this is the only latency the user actually experiences before text starts appearing, which makes it the number worth optimising for perceived speed.
In practice
TTFT is dominated by prefill, which is driven by input length. A 4,000-token prompt at 8,000 tokens per second prefill is half a second before generation even starts. That is why trimming input improves perceived speed as well as cost, and why prompt caching — which skips most of the prefill — helps latency more than people expect.
Common questions
How do I reduce time to first token?
Shorten the input and cache the stable prefix. TTFT is dominated by prefill, which scales with input length, so trimming the prompt improves felt speed and cost at the same time.
What is a good time to first token?
Under about a second feels responsive with streaming. Above two or three, users start to wonder whether anything is happening, and an interface cue matters more than the remaining milliseconds.