Provider rate limits compared: headers, tiers and 429s

A 429 looks like one error. In practice it is at least three: a rate window that will reopen in seconds, a quota or spend cap that will not reopen until a billing period ends, and a burst or acceleration limit that punishes how fast traffic arrived rather than how much. Each provider signals these differently, and a client that treats them all the same either gives up too early or retries into a wall.

This comparison covers what OpenAI, Anthropic, Azure OpenAI, the Gemini API and Amazon Bedrock count, how they scope limits, which headers report your standing, and what their 429s carry. It ends with retry code that branches on those differences.

What each provider counts

All five limit requests and tokens per unit of time, but the units and the scope differ enough to change how you design around them.

OpenAI measures requests and tokens per minute and per day, plus images per minute, and applies limits at the organization and project level, varying by model¹. Two details matter for capacity planning: the token count used for the limit is the larger of max_tokens and the estimated prompt size, and unsuccessful requests count toward the per-minute limit too¹.

Anthropic limits requests, input tokens and output tokens per minute per model, and uses a token bucket, so capacity refills continuously rather than resetting on a minute boundary². A limit stated per minute may be enforced over shorter intervals, so a burst can trip it while the minute's total looks fine. For most Claude models only uncached input tokens count toward the input limit, and max_tokens does not count toward the output limit, which is evaluated as tokens are produced².

Azure OpenAI assigns tokens per minute to each deployment out of a quota held per subscription, region, model and deployment type, and derives the requests-per-minute limit from it³. The token estimate is taken when the request arrives and includes the prompt and max_tokens. Request rate is evaluated over short windows, typically one or ten seconds, so a 600 RPM deployment can throttle more than ten requests in one second³.

The Gemini API measures requests per minute, input tokens per minute and requests per day, applied per project rather than per API key; the daily quota resets at midnight Pacific time⁴.

Amazon Bedrock applies tokens-per-minute quotas per model and per Region, with a separate quota for cross-Region inference profiles, a per-account tokens-per-day ceiling, and request-per-minute quotas for some models only⁵. Output tokens can burn quota at a multiple of their count: Bedrock documents a 5x, 10x or 15x output burndown for several Anthropic and OpenAI models on the bedrock-runtime endpoint⁶.

max_tokens is capacity you reserve

Three of the five providers reserve capacity from max_tokens before a single token is generated. OpenAI counts the larger of max_tokens and the prompt estimate¹; Azure OpenAI adds max_tokens to the estimate³; Bedrock deducts input plus max_tokens at the start of the request and returns the unused part at the end⁶. Setting max_tokens to the model maximum "just in case" therefore lowers how many requests can run at once, even though the bill only reflects tokens actually produced.

The practical rule: set max_tokens per workload from the observed output length distribution, with headroom, not from the model's ceiling. On Anthropic's direct API that rule matters for cost control rather than rate limits, since max_tokens does not factor into its output limit².

Reading your standing before the 429

Headers let a client slow down before it is throttled rather than after. OpenAI returns x-ratelimit-limit-requests, x-ratelimit-remaining-requests, x-ratelimit-reset-requests and the token equivalents, plus project-level token headers¹. Azure OpenAI returns the same x-ratelimit-* names, and a retry-after-ms on 429s³. Anthropic returns anthropic-ratelimit-requests-remaining, input and output token variants, and reset times in RFC 3339 format; its token headers show the most restrictive limit in effect, including a workspace limit². The Gemini rate-limit page and Bedrock's quota pages document no equivalent response headers⁴⁵, so for those two the 429 itself, and your own counters, are the signal.

A small normalizer turns those names into one number your client can act on:

/** Requests left in the current window, from whichever header family is present. */
export function requestsRemaining(headers: Headers): number | null {
  const candidates = [
    "x-ratelimit-remaining-requests",          // OpenAI, Azure OpenAI
    "anthropic-ratelimit-requests-remaining",  // Anthropic
    "x-ratelimit-remaining",                   // INFRO gateway keys (an upper bound)
  ];
  for (const name of candidates) {
    const raw = headers.get(name);
    if (raw !== null && /^\d+$/.test(raw.trim())) return Number(raw);
  }
  return null; // Gemini API, Bedrock: no documented header; rely on the 429
}

/** Slow down before the limit instead of absorbing 429s at it. */
export function shouldPace(headers: Headers, floor = 5): boolean {
  const left = requestsRemaining(headers);
  return left !== null && left <= floor;
}

Treat remaining counts as hints. Azure notes that enforcement across distributed infrastructure is not perfectly precise³, and a count read from one response is already stale by the next request.

Not every 429 is a rate limit

The same status code covers conditions with different cures.

OpenAI states it plainly: retrying billing, spend or quota errors will not restore access⁷. Anthropic notes that its SDKs' automatic retries fail too while a spend cap is in force². And Azure warns that many customers read capacity-related 429s as quota problems and request increases they do not need³. Read the body and the headers before deciding.

A retry policy that branches on the 429

Retry-After is either a number of seconds or an HTTP date¹¹; Azure adds a millisecond variant³. When the server gives a wait, use it. When it does not, use exponential backoff with full jitter, where each delay is drawn uniformly between zero and the capped exponential value; in AWS's simulations that variant did the least total client work of the jittered schemes¹². Either way, stop when the caller's deadline would be missed.

import email.utils
import random
import time

import httpx

RETRYABLE = {408, 429, 500, 502, 503, 504, 529}

# Codes that will not clear on a retry. Extend with your providers' own.
NON_TRANSIENT = {
    "enforced_spend_limit_reached",       # Anthropic tier spend cap (429, no retry-after)
    "organization_spend_limit_exceeded",  # OpenAI
    "project_spend_limit_exceeded",       # OpenAI
    "invalid_api_key_rate_limited",       # INFRO: fix the key first
    "no_provider_connected",              # INFRO 503: connect a provider
    "connection_out_of_scope",            # INFRO 503: fix the connection's scope
}


class GiveUp(Exception):
    pass


def server_wait(resp: httpx.Response) -> float | None:
    """Seconds the server asked for: retry-after-ms, then Retry-After (seconds or HTTP date)."""
    ms = resp.headers.get("retry-after-ms")
    if ms:
        try:
            return max(0.0, float(ms) / 1000)
        except ValueError:
            pass
    value = resp.headers.get("retry-after")
    if not value:
        return None
    try:
        return max(0.0, float(value))
    except ValueError:
        pass
    try:
        when = email.utils.parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    return max(0.0, when.timestamp() - time.time())


def error_code(resp: httpx.Response) -> str | None:
    try:
        err = resp.json().get("error")
    except ValueError:
        return None
    if not isinstance(err, dict):
        return None
    details = err.get("details")
    nested = details.get("error_code") if isinstance(details, dict) else None
    return err.get("code") or nested


def post_with_backoff(client: httpx.Client, url: str, payload: dict, *,
                      deadline_s: float = 30.0, base_s: float = 0.5,
                      cap_s: float = 8.0, max_attempts: int = 4) -> httpx.Response:
    start = time.monotonic()
    for attempt in range(max_attempts):
        remaining = deadline_s - (time.monotonic() - start)
        if remaining <= 0:
            break
        resp = None
        try:
            resp = client.post(url, json=payload, timeout=remaining)
        except httpx.TransportError:
            pass  # connection failure or timeout: retryable
        if resp is not None:
            if resp.status_code < 400:
                return resp
            if error_code(resp) in NON_TRANSIENT:
                raise GiveUp(f"{resp.status_code} {error_code(resp)}: needs a person, not a retry")
            if resp.status_code not in RETRYABLE:
                resp.raise_for_status()
        hint = server_wait(resp) if resp is not None else None
        if hint is not None:
            delay = hint + random.uniform(0, base_s)  # honor the wait, de-synchronize the fleet
        else:
            delay = random.uniform(0, min(cap_s, base_s * 2 ** attempt))  # full jitter
        if time.monotonic() - start + delay >= deadline_s:
            break  # the wait would outlive the caller
        time.sleep(delay)
    raise GiveUp("deadline or attempt budget exhausted")

Two cautions when you adopt this. First, the official SDKs already retry: Anthropic's retry transient failures twice by default and honor retry-after⁸, and Azure documents the same default for the OpenAI Python SDK³. Wrapping an SDK that retries in a loop that retries multiplies attempts, so set the SDK's max_retries to 0 when you own the policy. Second, a retried 429 is still traffic against the same allowance¹, so the fleet-wide effect of this loop is what matters, not one request's.

Capacity you can actually add

More keys rarely add capacity, because limits live at the organization, project, workspace, deployment or Region, not at the key. What does add capacity:

• A higher tier or quota. OpenAI graduates organizations automatically as spend grows¹; Anthropic moves organizations between tiers based on usage history²; Gemini tiers depend on billing setup, amount paid and days since first payment⁴; Azure and Bedrock take quota requests, and both give priority to customers already using their allocation³⁵.
• A second independent scope. Another Azure deployment or Region, another Bedrock Region or cross-Region profile, or a separate provider account that you hold, each with its own window.
• Fewer reserved tokens. A realistic max_tokens, and on Anthropic and Bedrock, prompt caching: cache reads do not count toward Anthropic's input limit for most models², nor toward Bedrock's token quotas⁶.
• A gentler ramp. OpenAI asks that traffic above one million input tokens per minute grow by no more than 50% every 15 minutes⁷; Anthropic recommends ramping gradually to avoid acceleration limits².

How INFRO handles a provider 429

INFRO routes requests over provider accounts you connect and does not add a shared pool of capacity: your accounts keep their own model, token and request quotas (limits). Two limits can therefore produce a 429 through it, and they are reported differently.

• Your INFRO key's own rate. Each key has a requests-per-minute tier, from 60 to 50,000 (200 by default), reported on every response in X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset. Remaining is an upper bound counted per serving location, so back off as it nears zero but treat the 429 and its Retry-After as authoritative.
• Your provider account throttling you. A provider 429 moves the request to the next of your connections that serves the model, or the next target on the route, and it never counts toward ejecting the connection (providers). If every connection tried fails and at least one was rate-limited, the response is 429 with code: "provider_rate_limited", and Retry-After carries the provider's own value, capped at 300 seconds, when it sent one (errors).

That second behaviour is only useful if the next connection has its own window. Two connections holding keys from the same OpenAI project share one allowance; two Azure deployments, two Bedrock Regions or accounts from different organizations do not. Each attempt, with its error and latency, is recorded on the request, so GET /v1/requests/{id} shows which connection was throttled and which one served. The retry code above works unchanged against INFRO: it already treats invalid_api_key_rate_limited, no_provider_connected and connection_out_of_scope as conditions to fix rather than retry.

Checklist

• Read each provider account's current limits from its console, and note the scope each limit applies to.
• Set max_tokens per workload from observed output lengths, not the model maximum.
• Parse Retry-After as seconds or an HTTP date, and retry-after-ms where Azure sends it.
• Branch on spend-cap and quota codes; stop retrying and alert on them.
• Use full jitter when no wait is given, and keep every retry inside the caller's deadline.
• Disable SDK retries when you wrap them in your own policy.
• Pace on remaining-quota headers where they exist; count your own traffic where they do not.
• Put failover capacity in a separate quota scope, not a second key in the same one.