# Load balancing LLM API keys: what adds capacity and what doesn't

URL: https://infro.io/blog/load-balance-llm-api-keys
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-20
Last modified: 2026-10-20
Last reviewed: 2026-10-20 (next review due 2026-11-19)
Topic: Routing and reliability
Format: Guide

> Adding API keys within one account does not raise your rate limits: OpenAI defines limits at the organization and project level, Anthropic at the organization level with optional workspace limits, and Gemini per project, not per API key. Real capacity comes from a higher usage tier, prompt caching (cached reads do not count toward Anthropic's input-token limits on most models), the batch APIs, provisioned throughput, and the same model on another provider. Spreading across keys is still useful for isolation, rotation and attribution.

Key takeaways:
- Keys inside one account share one limit: OpenAI limits organizations and projects, Anthropic organizations and workspaces, Gemini projects.
- Capacity comes from a higher tier, caching, batch APIs, provisioned throughput, and the same model on another provider account or cloud.
- On most Claude models, cached input reads do not count toward input-token rate limits, so caching raises effective throughput.
- Spread across keys for isolation, rotation and attribution, not for capacity.
- Balance on the provider's own headers (remaining requests and tokens), not round-robin.

**Do more API keys mean higher rate limits?** Usually not. Inside one account, every key draws on the same limit: OpenAI defines rate limits at the organization and project level[1], Anthropic at the organization level with optional per-workspace limits[2], and Google states that Gemini limits are "applied per project, not per API key"[3]. Round-robin across ten keys in one project gives you ten ways to hit one ceiling.

Capacity comes from elsewhere: tier, caching, batch, provisioned throughput, and the same model on another provider. Spreading across keys is still worth doing, for reasons that have nothing to do with throughput.

## Where the limit actually lives

| Provider | Unit the limit applies to | What raises it |
| --- | --- | --- |
| OpenAI | Organization and project | Usage tier (Free, Build, Launch, Grow), a separate project |
| Anthropic | Organization; workspaces can be set lower | Usage tier (Start, Build, Scale, Custom) |
| Google Gemini | Project, not key | Tier 1–3 by billing history, a separate project |

Table: From each provider's rate limit documentation, 2026-10-20.

Details that matter when you plan capacity:

- **OpenAI** tiers move automatically with your account's payment history; each tier also caps monthly spend. Responses carry `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and `x-ratelimit-reset-*` headers for requests and tokens, and `Retry-After` on a 429[1].
- **Anthropic** limits are per model class in requests, input tokens and output tokens per minute, enforced with a token bucket that refills continuously rather than resetting on the minute. Workspace limits can only partition the organization's limit; "organization-wide limits always apply, even if Workspace limits add up to more"[2].
- **Gemini** tiers depend on cumulative spend and time since first payment, and each paid tier also has a spend rate limit per ten minutes[3].

## What actually adds capacity

### 1. A higher tier

The boring answer, and the right first one. Every provider raises limits with account history, and each has a way to request more: Anthropic through **Request tier increase** in its console[2], OpenAI and Google through their tier systems. Ask before a launch, not during it.

### 2. Prompt caching (on Claude, it raises the limit itself)

Anthropic documents that for most Claude models only uncached input tokens count toward the input-tokens-per-minute limit: cache reads do not. Its example: with a 2,000,000 ITPM limit and an 80% cache hit rate, an organization can process 10,000,000 total input tokens per minute[2]. For agents and assistants with long, stable prefixes, caching is the largest capacity lever there is, and it lowers cost at the same time. [Prompt caching across providers](https://infro.io/blog/prompt-caching-break-even) covers when it pays.

Output limits behave differently: Anthropic evaluates output tokens per minute as they are generated, and `max_tokens` does not count against it[2].

### 3. Batch APIs

Batch endpoints have their own limits. Anthropic's Message Batches API has separate per-minute and queue limits shared across models[2]. Moving backfills, evaluations and nightly jobs to batch frees the interactive limit for users, and on most providers costs half as much.

### 4. The same model on another provider

The most effective capacity move for frontier models is serving the same model through a second provider account with its own quota: Claude on the Claude API and on Google Cloud; GPT models on OpenAI and Azure OpenAI. Each has independent limits, and you gain failover as a side effect. [Azure OpenAI and Bedrock behind one gateway](https://infro.io/blog/multi-cloud-azure-bedrock-one-gateway) covers the id and quota differences.

### 5. Separate projects, deliberately

Separate projects or workspaces per product or environment give each its own limits on providers that limit per project, and separate reporting. Do it to isolate workloads, not to multiply a quota you were assigned: read your provider's terms before you treat accounts or projects as a way around limits, and ask for an increase instead.

## What spreading across keys is good for

Multiple keys inside one account still earn their place:

- **Isolation.** A key per service or environment limits the blast radius of a leak to one revocation.
- **Rotation without downtime.** Two valid keys let you replace one while the other serves.
- **Attribution.** Provider usage reports can often be grouped by key or project; one key per workload makes that report useful.
- **Different accounts are different limits.** Keys from two distinct accounts — two clouds, or a direct contract and a cloud provider's deployment of the same model — do add capacity, because each carries its own limit.

## Sizing capacity before a launch

Work out the peak you need in the provider's own units before you choose between these levers:

1. **Requests per minute at peak.** Take your expected peak user actions per minute and multiply by the model calls each action makes. An agent step, a retrieval rerank and a final answer are three calls.
2. **Input and output tokens per minute.** Multiply by average input and output tokens per call. On Claude, split input into cached and uncached, because only uncached input counts toward the input limit on most models.
3. **Headroom.** Add at least 30% for retries and bursts; limits are enforced over short intervals, so a minute's average hides the second-by-second peak. Anthropic notes that a limit of 60 requests per minute may be enforced as one request per second[2].
4. **Compare with your tier.** If the peak is above your current limits, request an increase now, plan a second provider account for the same model, or move part of the load to batch.

> **Example: A support assistant at launch**
> Peak 600 conversations a minute, two Claude Sonnet 5.5 calls each: 1,200 requests a minute. Each call sends 20,000 input tokens, of which 18,000 are a cached prefix, and returns 500 output tokens. Uncached input is 1,200 × 2,000 = 2.4M tokens a minute; output is 0.6M. With 30% headroom that is about 1,560 requests, 3.1M input and 0.8M output tokens a minute: above the Start tier's Sonnet 5.5 limits of 1,000 requests, 2M input and 400K output tokens a minute, and inside the Build tier's 5,000, 5M and 1M[2]. Without caching, the input need would be ten times larger.

## Reading the headers to balance on headroom

Every major provider returns its remaining capacity on each response. OpenAI's headers include `x-ratelimit-remaining-requests` and `x-ratelimit-remaining-tokens`[1]; Anthropic's include `anthropic-ratelimit-requests-remaining`, `anthropic-ratelimit-input-tokens-remaining` and `anthropic-ratelimit-output-tokens-remaining`, with reset times in RFC 3339 format[2]. Record the latest values per target and prefer the one with the most room; a target whose remaining tokens fall below the size of the next request should not receive it.

## How to balance traffic well

Round-robin is the weakest strategy, because it ignores what each target has left. Better, in increasing order of effort:

1. **Priority with failover.** Send everything to the primary; move to the next target on a 429 or 5xx. Simple and predictable.
2. **Weighted.** Split by fixed weights when two accounts have known, stable capacity.
3. **Header-aware.** Read the provider's remaining-requests and remaining-tokens headers and prefer the target with the most headroom.
4. **Standby.** Keep one target at zero weight so it only takes traffic when the others fail.

```python
def pick(targets):
    """Prefer healthy targets with the most token headroom; standbys last."""
    healthy = [t for t in targets if not t.circuit_open]
    active = [t for t in healthy if not t.standby] or healthy
    return max(active, key=lambda t: t.remaining_tokens or 0)
```

Whatever the strategy, treat 429s and 5xx differently. A 429 means "this target is busy, try another or wait"; a run of 5xx means "this target is broken, stop sending to it". [Circuit breakers for LLM providers](https://infro.io/blog/llm-circuit-breakers-throttling) and [provider rate limits compared](https://infro.io/blog/llm-rate-limits-429-handling) go deeper.

## Spreading across keys with INFRO

INFRO lets you connect several keys for one provider, each as its own named connection — keys from separate accounts or projects, a direct contract beside a cloud deployment. A request that names a model directly is spread across that provider's keys, and a key marked standby only takes over when the others fail. A route can order targets by priority or split them by weight, with failure rules per kind of error, and a connection can list alternate hosts it tries before moving on. Every attempt is on the request's trace with the connection that made it. INFRO adds no capacity of its own: it holds no provider accounts, so the limits you route across are the limits your own accounts carry.

## Summary

Keys inside one account share one limit. Raise capacity with tiers, caching, batch and the same model on another provider account; use multiple keys for isolation, rotation and attribution; and balance on headroom and health rather than round-robin.

## Questions

### Do multiple OpenAI API keys increase my rate limit?

Not within one organization and project. OpenAI's guide says rate limits are defined at the organization level and the project level, so every key in the same project draws on the same limit. Capacity comes from a higher usage tier or a separate project with its own limits.

### Are Gemini rate limits per API key?

No. Google's rate limit page says rate limits are applied per project, not per API key. Creating more keys in one project does not raise any limit.

### How can I get more throughput from the Claude API?

Move to a higher usage tier, cache repeated prompt prefixes (cached reads do not count toward input-token rate limits on most Claude models), send non-urgent work through the Message Batches API, which has its own limits, and serve the same Claude model through another provider such as Google Cloud.

## Sources

1. Rate limits — OpenAI. https://developers.openai.com/api/docs/guides/rate-limits (accessed 2026-10-20)
2. Rate limits — Anthropic. https://platform.claude.com/docs/en/api/rate-limits (accessed 2026-10-20)
3. Rate limits — Google. https://ai.google.dev/gemini-api/docs/rate-limits (accessed 2026-10-20)