Load balancing LLM API keys: what adds capacity and what doesn't

Do more API keys mean higher rate limits? Usually not. Inside one account, every key draws on the same limit: OpenAI defines rate limits at the organization and project level¹, Anthropic at the organization level with optional per-workspace limits², and Google states that Gemini limits are "applied per project, not per API key"³. Round-robin across ten keys in one project gives you ten ways to hit one ceiling.

Capacity comes from elsewhere: tier, caching, batch, provisioned throughput, and the same model on another provider. Spreading across keys is still worth doing, for reasons that have nothing to do with throughput.

Where the limit actually lives

Details that matter when you plan capacity:

• OpenAI tiers move automatically with your account's payment history; each tier also caps monthly spend. Responses carry x-ratelimit-limit-*, x-ratelimit-remaining-* and x-ratelimit-reset-* headers for requests and tokens, and Retry-After on a 429¹.
• Anthropic limits are per model class in requests, input tokens and output tokens per minute, enforced with a token bucket that refills continuously rather than resetting on the minute. Workspace limits can only partition the organization's limit; "organization-wide limits always apply, even if Workspace limits add up to more"².
• Gemini tiers depend on cumulative spend and time since first payment, and each paid tier also has a spend rate limit per ten minutes³.

What actually adds capacity

1. A higher tier

The boring answer, and the right first one. Every provider raises limits with account history, and each has a way to request more: Anthropic through Request tier increase in its console², OpenAI and Google through their tier systems. Ask before a launch, not during it.

2. Prompt caching (on Claude, it raises the limit itself)

Anthropic documents that for most Claude models only uncached input tokens count toward the input-tokens-per-minute limit: cache reads do not. Its example: with a 2,000,000 ITPM limit and an 80% cache hit rate, an organization can process 10,000,000 total input tokens per minute². For agents and assistants with long, stable prefixes, caching is the largest capacity lever there is, and it lowers cost at the same time. Prompt caching across providers covers when it pays.

Output limits behave differently: Anthropic evaluates output tokens per minute as they are generated, and max_tokens does not count against it².

3. Batch APIs

Batch endpoints have their own limits. Anthropic's Message Batches API has separate per-minute and queue limits shared across models². Moving backfills, evaluations and nightly jobs to batch frees the interactive limit for users, and on most providers costs half as much.

4. The same model on another provider

The most effective capacity move for frontier models is serving the same model through a second provider account with its own quota: Claude on the Claude API and on Google Cloud; GPT models on OpenAI and Azure OpenAI. Each has independent limits, and you gain failover as a side effect. Azure OpenAI and Bedrock behind one gateway covers the id and quota differences.

5. Separate projects, deliberately

Separate projects or workspaces per product or environment give each its own limits on providers that limit per project, and separate reporting. Do it to isolate workloads, not to multiply a quota you were assigned: read your provider's terms before you treat accounts or projects as a way around limits, and ask for an increase instead.

What spreading across keys is good for

Multiple keys inside one account still earn their place:

• Isolation. A key per service or environment limits the blast radius of a leak to one revocation.
• Rotation without downtime. Two valid keys let you replace one while the other serves.
• Attribution. Provider usage reports can often be grouped by key or project; one key per workload makes that report useful.
• Different accounts are different limits. Keys from two distinct accounts — two clouds, or a direct contract and a cloud provider's deployment of the same model — do add capacity, because each carries its own limit.

Sizing capacity before a launch

Work out the peak you need in the provider's own units before you choose between these levers:

1. Requests per minute at peak. Take your expected peak user actions per minute and multiply by the model calls each action makes. An agent step, a retrieval rerank and a final answer are three calls.
2. Input and output tokens per minute. Multiply by average input and output tokens per call. On Claude, split input into cached and uncached, because only uncached input counts toward the input limit on most models.
3. Headroom. Add at least 30% for retries and bursts; limits are enforced over short intervals, so a minute's average hides the second-by-second peak. Anthropic notes that a limit of 60 requests per minute may be enforced as one request per second².
4. Compare with your tier. If the peak is above your current limits, request an increase now, plan a second provider account for the same model, or move part of the load to batch.

Reading the headers to balance on headroom

Every major provider returns its remaining capacity on each response. OpenAI's headers include x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens¹; Anthropic's include anthropic-ratelimit-requests-remaining, anthropic-ratelimit-input-tokens-remaining and anthropic-ratelimit-output-tokens-remaining, with reset times in RFC 3339 format². Record the latest values per target and prefer the one with the most room; a target whose remaining tokens fall below the size of the next request should not receive it.

How to balance traffic well

Round-robin is the weakest strategy, because it ignores what each target has left. Better, in increasing order of effort:

1. Priority with failover. Send everything to the primary; move to the next target on a 429 or 5xx. Simple and predictable.
2. Weighted. Split by fixed weights when two accounts have known, stable capacity.
3. Header-aware. Read the provider's remaining-requests and remaining-tokens headers and prefer the target with the most headroom.
4. Standby. Keep one target at zero weight so it only takes traffic when the others fail.

def pick(targets):
    """Prefer healthy targets with the most token headroom; standbys last."""
    healthy = [t for t in targets if not t.circuit_open]
    active = [t for t in healthy if not t.standby] or healthy
    return max(active, key=lambda t: t.remaining_tokens or 0)

Whatever the strategy, treat 429s and 5xx differently. A 429 means "this target is busy, try another or wait"; a run of 5xx means "this target is broken, stop sending to it". Circuit breakers for LLM providers and provider rate limits compared go deeper.

Spreading across keys with INFRO

INFRO lets you connect several keys for one provider, each as its own named connection — keys from separate accounts or projects, a direct contract beside a cloud deployment. A request that names a model directly is spread across that provider's keys, and a key marked standby only takes over when the others fail. A route can order targets by priority or split them by weight, with failure rules per kind of error, and a connection can list alternate hosts it tries before moving on. Every attempt is on the request's trace with the connection that made it. INFRO adds no capacity of its own: it holds no provider accounts, so the limits you route across are the limits your own accounts carry.

Summary

Keys inside one account share one limit. Raise capacity with tiers, caching, batch and the same model on another provider account; use multiple keys for isolation, rotation and attribution; and balance on headroom and health rather than round-robin.