# Provider rate limits compared: headers, tiers and 429s

URL: https://infro.io/blog/llm-rate-limits-429-handling
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Routing and reliability
Format: Comparison

> Every major provider limits requests and tokens per minute, but they count differently, scope limits differently and signal them differently: some send remaining-quota headers and a wait hint, some send neither. Treat a 429 with a wait hint as a timed pause, a 429 naming a spend cap or quota as a stop, and everything else with jittered backoff inside a deadline.

A 429 looks like one error. In practice it is at least three: a rate window that will reopen in seconds, a quota or spend cap that will not reopen until a billing period ends, and a burst or acceleration limit that punishes how fast traffic arrived rather than how much. Each provider signals these differently, and a client that treats them all the same either gives up too early or retries into a wall.

This comparison covers what OpenAI, Anthropic, Azure OpenAI, the Gemini API and Amazon Bedrock count, how they scope limits, which headers report your standing, and what their 429s carry. It ends with retry code that branches on those differences.

## What each provider counts

All five limit requests and tokens per unit of time, but the units and the scope differ enough to change how you design around them.

[Diagram: Limit dimensions, scope and the signals each provider documents, as published on 2026-09-29. Limit values are per account and change; read yours from the provider console or the response headers.]

**OpenAI** measures requests and tokens per minute and per day, plus images per minute, and applies limits at the organization and project level, varying by model[1]. Two details matter for capacity planning: the token count used for the limit is the larger of `max_tokens` and the estimated prompt size, and unsuccessful requests count toward the per-minute limit too[1].

**Anthropic** limits requests, input tokens and output tokens per minute per model, and uses a token bucket, so capacity refills continuously rather than resetting on a minute boundary[2]. A limit stated per minute may be enforced over shorter intervals, so a burst can trip it while the minute's total looks fine. For most Claude models only uncached input tokens count toward the input limit, and `max_tokens` does not count toward the output limit, which is evaluated as tokens are produced[2].

**Azure OpenAI** assigns tokens per minute to each deployment out of a quota held per subscription, region, model and deployment type, and derives the requests-per-minute limit from it[3]. The token estimate is taken when the request arrives and includes the prompt and `max_tokens`. Request rate is evaluated over short windows, typically one or ten seconds, so a 600 RPM deployment can throttle more than ten requests in one second[3].

**The Gemini API** measures requests per minute, input tokens per minute and requests per day, applied per project rather than per API key; the daily quota resets at midnight Pacific time[4].

**Amazon Bedrock** applies tokens-per-minute quotas per model and per Region, with a separate quota for cross-Region inference profiles, a per-account tokens-per-day ceiling, and request-per-minute quotas for some models only[5]. Output tokens can burn quota at a multiple of their count: Bedrock documents a 5x, 10x or 15x output burndown for several Anthropic and OpenAI models on the `bedrock-runtime` endpoint[6].

## max_tokens is capacity you reserve

Three of the five providers reserve capacity from `max_tokens` before a single token is generated. OpenAI counts the larger of `max_tokens` and the prompt estimate[1]; Azure OpenAI adds `max_tokens` to the estimate[3]; Bedrock deducts input plus `max_tokens` at the start of the request and returns the unused part at the end[6]. Setting `max_tokens` to the model maximum "just in case" therefore lowers how many requests can run at once, even though the bill only reflects tokens actually produced.

[Diagram: Amazon Bedrock's published example: the same request reserves 36,000 tokens of quota at the start with max_tokens at 32,000, and 5,250 with max_tokens at 1,250. Both settle at 9,000 once the response is done, because output burns at 5x on that model.]

The practical rule: set `max_tokens` per workload from the observed output length distribution, with headroom, not from the model's ceiling. On Anthropic's direct API that rule matters for cost control rather than rate limits, since `max_tokens` does not factor into its output limit[2].

## Reading your standing before the 429

Headers let a client slow down before it is throttled rather than after. OpenAI returns `x-ratelimit-limit-requests`, `x-ratelimit-remaining-requests`, `x-ratelimit-reset-requests` and the token equivalents, plus project-level token headers[1]. Azure OpenAI returns the same `x-ratelimit-*` names, and a `retry-after-ms` on 429s[3]. Anthropic returns `anthropic-ratelimit-requests-remaining`, input and output token variants, and reset times in RFC 3339 format; its token headers show the most restrictive limit in effect, including a workspace limit[2]. The Gemini rate-limit page and Bedrock's quota pages document no equivalent response headers[4][5], so for those two the 429 itself, and your own counters, are the signal.

A small normalizer turns those names into one number your client can act on:

```typescript
/** Requests left in the current window, from whichever header family is present. */
export function requestsRemaining(headers: Headers): number | null {
  const candidates = [
    "x-ratelimit-remaining-requests",          // OpenAI, Azure OpenAI
    "anthropic-ratelimit-requests-remaining",  // Anthropic
    "x-ratelimit-remaining",                   // INFRO gateway keys (an upper bound)
  ];
  for (const name of candidates) {
    const raw = headers.get(name);
    if (raw !== null && /^\d+$/.test(raw.trim())) return Number(raw);
  }
  return null; // Gemini API, Bedrock: no documented header; rely on the 429
}

/** Slow down before the limit instead of absorbing 429s at it. */
export function shouldPace(headers: Headers, floor = 5): boolean {
  const left = requestsRemaining(headers);
  return left !== null && left <= floor;
}
```

Treat remaining counts as hints. Azure notes that enforcement across distributed infrastructure is not perfectly precise[3], and a count read from one response is already stale by the next request.

## Not every 429 is a rate limit

The same status code covers conditions with different cures.

[Diagram: Four kinds of 429 and the response each needs. The branch that matters most is the second: a spend cap or quota does not clear on a retry.]

| Provider | Rate window | Cap or quota that will not clear on retry | Other |
| --- | --- | --- | --- |
| OpenAI | 429 with `Retry-After` when present[7] | 429 with `organization_spend_limit_exceeded`, `project_spend_limit_exceeded` and related billing codes[7] | 429 when traffic ramps faster than the service can handle[7] |
| Anthropic | 429 `rate_limit_error` with `retry-after`[2] | 429 with `error_code: enforced_spend_limit_reached` and no `retry-after`[2] | Acceleration limits on sharp usage increases; 529 `overloaded_error` for API-wide load[8] |
| Azure OpenAI | 429 with `retry-after-ms`[3] | None documented; a deployment whose TPM allocation is too small keeps throttling until quota is reallocated[3] | Capacity throttling and temporary limit reductions on shared Standard deployments[3] |
| Gemini API | 429 for per-minute limits[9] | 429 for an exhausted daily quota[9] | 429 `RESOURCE_EXHAUSTED` when spend-based limits are exceeded[4] |
| Amazon Bedrock | `ThrottlingException`, HTTP 429[10] | `ServiceQuotaExceededException`, HTTP 400[10] | `ModelNotReadyException`, also 429, retried by the AWS SDK[10] |

Table: What a throttling response means per provider, from each provider's documentation on 2026-09-29.

OpenAI states it plainly: retrying billing, spend or quota errors will not restore access[7]. Anthropic notes that its SDKs' automatic retries fail too while a spend cap is in force[2]. And Azure warns that many customers read capacity-related 429s as quota problems and request increases they do not need[3]. Read the body and the headers before deciding.

## A retry policy that branches on the 429

`Retry-After` is either a number of seconds or an HTTP date[11]; Azure adds a millisecond variant[3]. When the server gives a wait, use it. When it does not, use exponential backoff with full jitter, where each delay is drawn uniformly between zero and the capped exponential value; in AWS's simulations that variant did the least total client work of the jittered schemes[12]. Either way, stop when the caller's deadline would be missed.

```python
import email.utils
import random
import time

import httpx

RETRYABLE = {408, 429, 500, 502, 503, 504, 529}

# Codes that will not clear on a retry. Extend with your providers' own.
NON_TRANSIENT = {
    "enforced_spend_limit_reached",       # Anthropic tier spend cap (429, no retry-after)
    "organization_spend_limit_exceeded",  # OpenAI
    "project_spend_limit_exceeded",       # OpenAI
    "invalid_api_key_rate_limited",       # INFRO: fix the key first
    "no_provider_connected",              # INFRO 503: connect a provider
    "connection_out_of_scope",            # INFRO 503: fix the connection's scope
}


class GiveUp(Exception):
    pass


def server_wait(resp: httpx.Response) -> float | None:
    """Seconds the server asked for: retry-after-ms, then Retry-After (seconds or HTTP date)."""
    ms = resp.headers.get("retry-after-ms")
    if ms:
        try:
            return max(0.0, float(ms) / 1000)
        except ValueError:
            pass
    value = resp.headers.get("retry-after")
    if not value:
        return None
    try:
        return max(0.0, float(value))
    except ValueError:
        pass
    try:
        when = email.utils.parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    return max(0.0, when.timestamp() - time.time())


def error_code(resp: httpx.Response) -> str | None:
    try:
        err = resp.json().get("error")
    except ValueError:
        return None
    if not isinstance(err, dict):
        return None
    details = err.get("details")
    nested = details.get("error_code") if isinstance(details, dict) else None
    return err.get("code") or nested


def post_with_backoff(client: httpx.Client, url: str, payload: dict, *,
                      deadline_s: float = 30.0, base_s: float = 0.5,
                      cap_s: float = 8.0, max_attempts: int = 4) -> httpx.Response:
    start = time.monotonic()
    for attempt in range(max_attempts):
        remaining = deadline_s - (time.monotonic() - start)
        if remaining <= 0:
            break
        resp = None
        try:
            resp = client.post(url, json=payload, timeout=remaining)
        except httpx.TransportError:
            pass  # connection failure or timeout: retryable
        if resp is not None:
            if resp.status_code < 400:
                return resp
            if error_code(resp) in NON_TRANSIENT:
                raise GiveUp(f"{resp.status_code} {error_code(resp)}: needs a person, not a retry")
            if resp.status_code not in RETRYABLE:
                resp.raise_for_status()
        hint = server_wait(resp) if resp is not None else None
        if hint is not None:
            delay = hint + random.uniform(0, base_s)  # honor the wait, de-synchronize the fleet
        else:
            delay = random.uniform(0, min(cap_s, base_s * 2 ** attempt))  # full jitter
        if time.monotonic() - start + delay >= deadline_s:
            break  # the wait would outlive the caller
        time.sleep(delay)
    raise GiveUp("deadline or attempt budget exhausted")
```

Two cautions when you adopt this. First, the official SDKs already retry: Anthropic's retry transient failures twice by default and honor `retry-after`[8], and Azure documents the same default for the OpenAI Python SDK[3]. Wrapping an SDK that retries in a loop that retries multiplies attempts, so set the SDK's `max_retries` to 0 when you own the policy. Second, a retried 429 is still traffic against the same allowance[1], so the fleet-wide effect of this loop is what matters, not one request's.

## Capacity you can actually add

More keys rarely add capacity, because limits live at the organization, project, workspace, deployment or Region, not at the key. What does add capacity:

- **A higher tier or quota.** OpenAI graduates organizations automatically as spend grows[1]; Anthropic moves organizations between tiers based on usage history[2]; Gemini tiers depend on billing setup, amount paid and days since first payment[4]; Azure and Bedrock take quota requests, and both give priority to customers already using their allocation[3][5].
- **A second independent scope.** Another Azure deployment or Region, another Bedrock Region or cross-Region profile, or a separate provider account that you hold, each with its own window.
- **Fewer reserved tokens.** A realistic `max_tokens`, and on Anthropic and Bedrock, prompt caching: cache reads do not count toward Anthropic's input limit for most models[2], nor toward Bedrock's token quotas[6].
- **A gentler ramp.** OpenAI asks that traffic above one million input tokens per minute grow by no more than 50% every 15 minutes[7]; Anthropic recommends ramping gradually to avoid acceleration limits[2].

## How INFRO handles a provider 429

INFRO routes requests over provider accounts you connect and does not add a shared pool of capacity: your accounts keep their own model, token and request quotas ([limits](https://infro.io/docs/api/limits)). Two limits can therefore produce a 429 through it, and they are reported differently.

- **Your INFRO key's own rate.** Each key has a requests-per-minute tier, from 60 to 50,000 (200 by default), reported on every response in `X-RateLimit-Limit`, `X-RateLimit-Remaining` and `X-RateLimit-Reset`. `Remaining` is an upper bound counted per serving location, so back off as it nears zero but treat the 429 and its `Retry-After` as authoritative.
- **Your provider account throttling you.** A provider 429 moves the request to the next of your connections that serves the model, or the next target on the [route](https://infro.io/docs/routes), and it never counts toward ejecting the connection ([providers](https://infro.io/docs/providers)). If every connection tried fails and at least one was rate-limited, the response is `429` with `code: "provider_rate_limited"`, and `Retry-After` carries the provider's own value, capped at 300 seconds, when it sent one ([errors](https://infro.io/docs/api/errors)).

That second behaviour is only useful if the next connection has its own window. Two connections holding keys from the same OpenAI project share one allowance; two Azure deployments, two Bedrock Regions or accounts from different organizations do not. Each attempt, with its error and latency, is recorded on the request, so `GET /v1/requests/{id}` shows which connection was throttled and which one served. The retry code above works unchanged against INFRO: it already treats `invalid_api_key_rate_limited`, `no_provider_connected` and `connection_out_of_scope` as conditions to fix rather than retry.

## Checklist

- [ ] Read each provider account's current limits from its console, and note the scope each limit applies to.
- [ ] Set `max_tokens` per workload from observed output lengths, not the model maximum.
- [ ] Parse `Retry-After` as seconds or an HTTP date, and `retry-after-ms` where Azure sends it.
- [ ] Branch on spend-cap and quota codes; stop retrying and alert on them.
- [ ] Use full jitter when no wait is given, and keep every retry inside the caller's deadline.
- [ ] Disable SDK retries when you wrap them in your own policy.
- [ ] Pace on remaining-quota headers where they exist; count your own traffic where they do not.
- [ ] Put failover capacity in a separate quota scope, not a second key in the same one.

## Questions

### Does a request that fails with 429 still count against my rate limit?

It can. OpenAI and Azure OpenAI both state that unsuccessful requests count toward the per-minute limit, which is why retrying in a tight loop keeps an account throttled for longer.

### Do two API keys on the same provider account double my rate limit?

Usually not. OpenAI applies limits to the organization and project, Anthropic to the organization and optional workspaces, and Gemini to the project, so extra keys in the same scope share one allowance.

## Sources

1. Rate limits — OpenAI. https://developers.openai.com/api/docs/guides/rate-limits (accessed 2026-09-29)
2. Rate limits — Anthropic. https://platform.claude.com/docs/en/api/rate-limits (accessed 2026-09-29)
3. Manage Azure OpenAI in Microsoft Foundry Models quota — Microsoft. https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/quota (accessed 2026-09-29)
4. Rate limits — Google AI for Developers. https://ai.google.dev/gemini-api/docs/rate-limits (accessed 2026-09-29)
5. Quotas for the bedrock-runtime endpoint — Amazon Web Services. https://docs.aws.amazon.com/bedrock/latest/userguide/quotas-runtime.html (accessed 2026-09-29)
6. How tokens are counted in Amazon Bedrock — Amazon Web Services. https://docs.aws.amazon.com/bedrock/latest/userguide/quotas-token-burndown.html (accessed 2026-09-29)
7. Error codes — OpenAI. https://developers.openai.com/api/docs/guides/error-codes (accessed 2026-09-29)
8. Claude API errors — Anthropic. https://platform.claude.com/docs/en/api/errors (accessed 2026-09-29)
9. API errors — Google AI for Developers. https://ai.google.dev/gemini-api/docs/api-errors (accessed 2026-09-29)
10. InvokeModel — Amazon Bedrock API Reference — Amazon Web Services. https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InvokeModel.html (accessed 2026-09-29)
11. RFC 9110: HTTP Semantics — Retry-After — IETF. https://httpwg.org/specs/rfc9110.html#field.retry-after (accessed 2026-09-29)
12. Exponential Backoff And Jitter — AWS Architecture Blog. https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ (accessed 2026-09-29)