TokenPad

Recipe · Reliability

Retry with exponential backoff

Handle 429 and 529 correctly: which errors to retry, why jitter matters, and the one 429 you must never retry.

The problem

Rate limits and capacity errors are normal operating conditions on LLM APIs, not exceptional ones. Without retry logic your application fails on transient conditions that clear in seconds.

With naive retry logic it fails worse: every concurrent worker retries at the same instant and turns a brief overage into a sustained outage.

Backoff with jitter, and a retry decisiontypescript
interface RetryOptions {
  maxAttempts?: number;
  baseDelayMs?: number;
  maxDelayMs?: number;
}

/** Not every 429 should be retried — insufficient_quota never clears on its own. */
function isRetryable(error: unknown): boolean {
  const status = (error as { status?: number })?.status;
  const code = (error as { code?: string })?.code;

  if (code === 'insufficient_quota') return false; // billing, not throughput
  if (status === 429) return true;                 // rate limit
  if (status === 408) return true;                 // timeout
  if (status && status >= 500) return true;        // 500, 503, 529 overloaded
  return false;
}

export async function withRetry<T>(
  fn: () => Promise<T>,
  { maxAttempts = 5, baseDelayMs = 500, maxDelayMs = 30_000 }: RetryOptions = {},
): Promise<T> {
  let lastError: unknown;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (error) {
      lastError = error;
      if (!isRetryable(error) || attempt === maxAttempts - 1) throw error;

      // Exponential, capped, with full jitter so concurrent workers spread out
      // instead of retrying in lockstep and re-creating the overload.
      const ceiling = Math.min(baseDelayMs * 2 ** attempt, maxDelayMs);
      const delay = Math.random() * ceiling;

      // Honour the provider's own hint when it sends one.
      const retryAfter = Number((error as { headers?: Record<string, string> })?.headers?.['retry-after']);
      const wait = Number.isFinite(retryAfter) ? retryAfter * 1000 : delay;

      await new Promise((resolve) => setTimeout(resolve, wait));
    }
  }

  throw lastError;
}
The same thing in Pythonpython
import random
import time
from typing import Callable, TypeVar

T = TypeVar("T")

NON_RETRYABLE_CODES = {"insufficient_quota", "invalid_api_key", "model_not_found"}


def is_retryable(error: Exception) -> bool:
    code = getattr(error, "code", None)
    if code in NON_RETRYABLE_CODES:
        return False
    status = getattr(error, "status_code", None)
    return status in (408, 429) or (status is not None and status >= 500)


def with_retry(fn: Callable[[], T], max_attempts: int = 5, base_delay: float = 0.5) -> T:
    last_error: Exception | None = None

    for attempt in range(max_attempts):
        try:
            return fn()
        except Exception as error:
            last_error = error
            if not is_retryable(error) or attempt == max_attempts - 1:
                raise

            ceiling = min(base_delay * (2 ** attempt), 30.0)
            time.sleep(random.uniform(0, ceiling))

    raise last_error  # unreachable, but keeps the type checker honest

Why it is written this way

Full jitter, not fixed delay

A fixed delay makes every worker retry at the same moment, which reproduces the overload that caused the error. Randomising the wait across the whole window spreads the load and is what makes backoff actually work.

Honour retry-after when present

Providers frequently send a header telling you exactly how long to wait. Using it beats guessing, and ignoring it while retrying aggressively is how accounts get throttled harder.

A 429 is two different errors

A rate limit clears in seconds and should be retried. An insufficient_quota means you are out of credit and will never clear — retrying it fills your logs and delays the alert you actually need.

What breaks the naive version

  • Retrying insufficient_quota. It shares the 429 status with rate limiting and needs the opposite handling.
  • Retrying a 400. A malformed request will be malformed on every attempt.
  • No maximum. Unbounded retries turn a provider incident into an unbounded bill and a hanging request.

Check your numbers