LLM API error
Response cut off mid-sentence
The model stopped because it hit a limit, not because it finished. Why there is no error, how to detect it, and how to size the ceiling properly.
The message you are seeing
"finish_reason": "length"
"stop_reason": "max_tokens"
What it means
Generation stopped because a limit was reached rather than because the model finished. There is no error — the request succeeded, the response is just incomplete.
This is the failure mode that reaches users, because nothing in your logs marks it as a problem unless you check the stop reason explicitly.
Why it happens
Most likely cause first.
- 1
max_tokens set too low for the task
Usually a value copied from an example and never revisited. It was fine until someone asked a question that needed a longer answer.
- 2
Input plus max_tokens exceeds the context window
On newer models the request is accepted and generation stops when the window is reached, rather than failing up front. Harder to notice than a rejection.
- 3
A reasoning model spent the budget thinking
Internal reasoning is billed as output and counts towards max_tokens. A ceiling sized for the visible answer can be exhausted before the answer starts.
- 4
A stop sequence matched inside a valid answer
A double newline as a stop sequence works for single-paragraph answers and truncates every multi-paragraph one.
How to fix it
Check the stop reason on every response
It distinguishes a finished answer from a truncated one. This is a few lines of code and turns a silent product defect into a handled case.
Size max_tokens from your data
Take the 95th percentile of observed answer lengths and add about thirty percent. Setting it to the model maximum reserves window you could have used and, on some providers, quota you never spend.
max_tokens PlannerA ceiling derived from your p95, checked against window and quota.Confirm the input leaves room
Input plus the ceiling must fit inside the window. Decide the output reserve first and treat the remainder as your real input capacity.
Context Window Budget PlannerWindow allocation with a safety margin and the fixed overhead shown.Test your stop sequences against real output
Verify the sequence is one the model produces at the end and never inside a valid answer.
Stop Sequence TesterWhere the model stops, what survives, and what you paid for regardless.
Stopping it happening again
- Treat a truncated response as an error in your own code: retry with a higher ceiling, or tell the user, but never render it as if it were complete.
- For structured output, validate that the JSON parses before using it — truncation produces syntactically invalid output, not a shorter valid object.
- On reasoning models, size max_tokens for thinking plus answer rather than for the answer alone.
Tools that help
Frequently asked questions
- Why is there no error when a response is truncated?
- Because the request succeeded. The model generated tokens up to the limit you set, which is exactly what you asked for. The stop reason field is how the API tells you it was cut short.
- Should I just set max_tokens to the maximum?
- No. It reserves context window you could have retrieved into, and on providers that count the ceiling against your rate limit it consumes throughput you never use. Size it from your observed answer lengths.