Parameters
max_tokens
max_tokens caps how many tokens the model may generate in its reply. It is a ceiling on output only, subtracted from the same context window the input occupies.
Setting it to the model maximum "to be safe" is the common default and the wrong one: it reserves window you could have retrieved into, and on providers that count the ceiling against your rate limit it consumes quota you never use.
Derive it from the 95th percentile of your observed answer lengths with a margin of around thirty percent.
In practice
Left unset, output length is bounded only by the model, and one prompt that induces a rambling answer costs what thousands of normal ones do. Set it from the longest legitimate answer, then handle the truncation case — because once the limit exists, hitting it returns HTTP 200 with a half-finished answer and nothing in the transport says so.
Common questions
What should I set max_tokens to?
The longest legitimate answer plus a small margin, measured rather than guessed. Set too low it truncates good answers; left unset, output length is bounded only by the model and so is your bill.
What happens when max_tokens is reached?
The response returns successfully with an answer cut off mid-sentence, and only the stop reason indicates it. Code that does not check that field treats a half-finished answer as a complete one.