TokenPad

LLM API error

Request timed out

Timeouts on LLM APIs are usually about output length, not network conditions. Why default client timeouts are wrong, and what to set instead.

The message you are seeing

Client library
APITimeoutError: Request timed out.
Gateway
504 Gateway Timeout

What it means

Your client gave up waiting. Generation is sequential — one token at a time — so a long response legitimately takes a long time, and default HTTP timeouts designed for ordinary APIs are frequently too short.

A request generating two thousand tokens at fifty tokens per second takes forty seconds before network overhead. Many clients default to thirty.

Why it happens

Most likely cause first.

  1. 1

    Default client timeout too low for the output length

    The most common cause by a distance. It works in testing with short answers and fails on the first long one.

  2. 2

    A very long prompt slowing time to first token

    The model must process your entire input before generating anything. On a large prompt that prefill alone can consume most of a short timeout.

  3. 3

    A reasoning model thinking for a long time

    Internal reasoning happens before any visible output. At high effort the wait before the first token can be substantial.

  4. 4

    An infrastructure timeout you did not set

    A load balancer, API gateway or serverless function limit cutting the connection independently of your client configuration.

How to fix it

  1. Work out how long the request should legitimately take

    Time to first token plus output tokens divided by generation speed. If your timeout is below that, the timeout is the bug.

    LLM Pipeline Latency EstimatorMulti-step latency with serial and parallel totals compared.

  2. Raise the client timeout deliberately

    Set it from the expected duration with margin, not to an arbitrary large number. A timeout that never fires is not a timeout.

  3. Stream the response

    Streaming keeps the connection active and, more importantly, means the user sees text after the first token rather than after the last. It converts a timeout risk into a progress indicator.

    Streaming Latency SimulatorPrefill, first token, generation rate — and the perceived latency gain.

  4. Shorten the input to cut time to first token

    Prefill is often the largest component of the initial wait. Reducing prompt size improves latency as well as cost.

    Prompt Token OptimizerCuts whitespace, minifies JSON, collapses blank lines. Shows tokens saved and the annual value.

Stopping it happening again

  • Check every timeout in the path, not just your client — gateways and serverless platforms impose their own and will cut the connection regardless of your setting.
  • Use streaming for anything user-facing. It removes most timeout exposure and dramatically improves perceived speed.
  • Cap max_tokens realistically so a runaway generation cannot run until something times out.

Tools that help

Frequently asked questions

What timeout should I set?
Derive it: time to first token plus your 95th percentile output length divided by the model’s generation speed, plus margin. For most chat workloads that lands between sixty and a hundred and twenty seconds, which is far above typical defaults.
Does streaming avoid timeouts entirely?
It avoids most of them, because data flows continuously and idle timeouts never trigger. You still need a sensible total limit so a stalled generation does not hang forever.