# Circuit breakers for LLM providers: throttling is not failure

URL: https://infro.io/blog/llm-circuit-breakers-throttling
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Routing and reliability
Format: Analysis

> A circuit breaker for an LLM provider should open on availability failures — timeouts, connection errors, 5xx and overload — and on rejected credentials, but never on rate-limit 429s, which mean the account is healthy and busy. Handle throttling with a separate gate that honors the provider's wait hint and sends traffic to another quota scope, and park an account whose spend cap or quota is exhausted until it resets.

A circuit breaker has one job: stop sending requests to something that is failing, so callers fail fast instead of waiting, and the failing thing gets room to recover. Fowler's description is the one most implementations follow — a closed breaker passes calls and counts failures, an open one returns an error immediately, and after a timeout a half-open breaker lets a trial call through to see whether the fault has cleared[1].

The pattern assumes an error means the dependency is unwell. For LLM providers that assumption breaks on the most common error of all.

## The 429 problem

A 429 from a model provider usually means the opposite of unwell. The account answered, promptly, and said: this account has used its allowance for now. Anthropic's limits refill continuously as a token bucket[2]; OpenAI's are windows per organization, project and model[3]. In both cases, capacity returns on its own within seconds to a minute, and nothing is broken.

Count those 429s as failures and three things go wrong.

- **A busy account is ejected for being busy.** The breaker opens exactly when traffic is highest, removing capacity at peak.
- **The ejection outlasts the throttle.** A rate window reopens in seconds; a breaker cool-off is typically tens of seconds to minutes. The breaker turns a short slowdown into a longer outage of that account.
- **It cascades.** Traffic moves to the next account, which is now over its limit too, whose breaker opens, and so on. With every account open, the application fails requests that the providers would have served a few seconds later.

Hammering is no better than ejecting. OpenAI and Azure both state that unsuccessful requests count toward the per-minute limit[3][4], so a client that keeps retrying a throttled account keeps it throttled.

## Three conditions, three mechanisms

The fix is to stop asking one mechanism to handle three different conditions.

[Diagram: What each signal means for an account, and whether it should count toward opening a breaker. Only availability failures and rejected credentials do.]

1. **Availability failure** — timeouts, connection errors, 5xx, and overload errors such as Anthropic's `529 overloaded_error`[5]. This is what the breaker is for. Count it, open on a threshold, cool off, probe.
2. **Throttling** — a 429 with a wait hint. Handle it with a *throttle gate*: mark the account unavailable until the time the provider named, route around it meanwhile, and let it back without a probe when the time passes. Azure's guidance calls this accelerated circuit breaking: when a failure response itself says how long to wait, the breaker can trip immediately for that long[6].
3. **Quota or spend exhaustion** — a 429 or 400 that names a cap. Anthropic's tier spend cap, for example, returns 429 with no `retry-after` and keeps failing until access resumes[2]. *Park* the account until the period resets and tell a person; neither a probe nor a timer will fix it.

Credential rejections belong with availability, but deserve a lower threshold and a page: an account whose key was revoked will not recover by waiting. Invalid requests — a 400 for a malformed body — say nothing about the account at all.

[Diagram: A breaker that counts availability failures only, with throttling and quota handled beside it rather than inside it.]

## Scope each breaker to one quota and one failure domain

Microsoft's pattern notes warn against a single breaker for a resource that has several independent providers behind it: one healthy part gets blocked while a failing one keeps receiving traffic[6]. For LLM traffic the independent parts are provider accounts, deployments and regions, and often models, since rate limits are applied per model[3][2].

- Key the **throttle gate** by the quota scope: the account and model, or the Azure deployment.
- Key the **availability breaker** by the failure domain: the account, deployment or region endpoint.
- Key **quota parking** by whatever the cap applies to — usually the whole account or project.

Not every overload is local. Anthropic describes 529 as API-wide load and separately documents acceleration limits that return 429 when an organization's usage rises sharply[5]. Azure documents capacity throttling and temporary rate-limit reductions on shared deployments, and warns that these are often misread as quota problems[4]. When every account at one provider trips together, the useful failover target is a different provider or region, not a second key.

## A breaker that tells the difference

The class below keeps the three mechanisms separate. It opens on a failure rate over a rolling window, lengthens the open period each time it reopens, allows one trial at a time when half-open, and logs every state change, which Fowler singles out as the most useful signal a breaker produces[1].

```typescript
export type Verdict = "success" | "availability" | "credential" | "throttled" | "quota" | "request";

const QUOTA_CODES = new Set([
  "enforced_spend_limit_reached",       // Anthropic tier spend cap
  "organization_spend_limit_exceeded",  // OpenAI
  "project_spend_limit_exceeded",       // OpenAI
]);

export function classify(status: number | null, body: unknown): Verdict {
  if (status === null) return "availability";                // connection error or timeout
  if (status < 400) return "success";
  const err = (body as { error?: { code?: string; details?: { error_code?: string } } })?.error;
  const code = err?.code ?? err?.details?.error_code ?? null;
  if (code && QUOTA_CODES.has(code)) return "quota";
  if (status === 429) return "throttled";
  if (status === 401) return "credential";
  if (status === 403) return "credential";                   // unless you can identify a moderation refusal
  if (status === 408 || status >= 500) return "availability"; // includes 529
  return "request";                                          // 400, 404, 422: not the account's fault
}

type State = "closed" | "open" | "half_open";

export class ProviderBreaker {
  private state: State = "closed";
  private window: boolean[] = [];        // true = availability failure
  private credentialFailures = 0;
  private opens = 0;
  private openUntil = 0;
  private probeInFlight = false;
  private throttledUntil = 0;
  private parkedUntil = 0;

  constructor(
    readonly name: string,
    private readonly o = {
      windowSize: 20, minCalls: 10, failureRate: 0.5, credentialTrips: 2,
      baseOpenMs: 30_000, maxOpenMs: 15 * 60_000, defaultThrottleMs: 5_000,
    },
    private readonly log: (msg: string) => void = console.log,
  ) {}

  /** May a request go to this account now? Call immediately before sending. */
  canSend(now = Date.now()): boolean {
    if (now < this.parkedUntil || now < this.throttledUntil) return false;
    if (this.state === "open") {
      if (now < this.openUntil) return false;
      this.transition("half_open");
    }
    if (this.state === "half_open") {
      if (this.probeInFlight) return false;
      this.probeInFlight = true;                  // exactly one trial at a time
    }
    return true;
  }

  record(v: Verdict, hint: { retryAfterMs?: number | null; resetsAt?: number } = {}, now = Date.now()) {
    this.probeInFlight = false;
    switch (v) {
      case "success":
        return this.healthy();
      case "throttled":                           // healthy and busy: gate, do not count
        this.throttledUntil = now + (hint.retryAfterMs ?? this.o.defaultThrottleMs);
        return this.healthy();
      case "quota":
        this.parkedUntil = hint.resetsAt ?? now + 60 * 60_000;
        this.log(`${this.name}: parked until ${new Date(this.parkedUntil).toISOString()} (quota or spend cap)`);
        return;
      case "request":
        return;                                   // says nothing about the account
      case "credential":
        this.credentialFailures += 1;
        if (this.credentialFailures >= this.o.credentialTrips) return this.trip(now);
        return this.fail(now);
      case "availability":
        return this.fail(now);
    }
  }

  private healthy() {
    this.credentialFailures = 0;
    this.push(false);
    if (this.state === "half_open") {
      this.opens = 0;
      this.window = [];
      this.transition("closed");
    }
  }

  private fail(now: number) {
    if (this.state === "half_open") return this.trip(now);   // failed trial: reopen, longer
    this.push(true);
    const failures = this.window.filter(Boolean).length;
    if (this.window.length >= this.o.minCalls && failures / this.window.length >= this.o.failureRate) {
      this.trip(now);
    }
  }

  private trip(now: number) {
    this.opens += 1;
    const ms = Math.min(this.o.maxOpenMs, this.o.baseOpenMs * 2 ** (this.opens - 1));
    this.openUntil = now + ms;
    this.window = [];
    this.transition("open", `for ${Math.round(ms / 1000)}s`);
  }

  private push(failed: boolean) {
    this.window.push(failed);
    if (this.window.length > this.o.windowSize) this.window.shift();
  }

  private transition(to: State, detail = "") {
    if (this.state === to) return;
    this.log(`${this.name}: ${this.state} -> ${to} ${detail}`.trim());
    this.state = to;
  }
}
```

A router then takes the first account whose breaker says `canSend()`, calls it, and passes the verdict back with `record()`, including the parsed `Retry-After` for a throttle. When no account can send, fail fast with an error your application handles, rather than queueing blind. And keep retries bounded independently of the breaker: Google's SRE book describes a per-client retry budget that stops retrying once retries exceed about 10% of requests, which caps the extra load a fleet can add during an incident[7].

## Tuning without guessing

Microsoft recommends matching the breaker to the likely recovery pattern: open too long and it rejects requests after the fault has cleared; move to half-open too soon and it flaps[6]. Fowler adds that thresholds can differ by error type, for example tolerating more timeouts than connection failures[1]. For provider accounts, a sensible starting point is a short first cool-off that grows while failures continue, a failure-rate threshold with a minimum number of calls so that two early errors cannot trip it, and a much lower threshold for rejected credentials. Then verify it with a drill rather than trusting the numbers.

## How INFRO takes a connection out of rotation

INFRO applies this separation to each provider connection you connect, per connection rather than per provider kind, so a staging key failing does not eject a production key ([providers](https://infro.io/docs/providers)).

[Diagram: The connection health rules INFRO documents: five consecutive availability failures or two rejected credentials eject a connection; the cool-off starts at 30 seconds and grows to at most 15 minutes; one success clears it; a 429 never counts.]

- **Availability failures and credential rejections count.** Five in a row eject the connection for availability errors, two for a rejected credential. The cool-off starts at 30 seconds and grows while failures continue, to at most 15 minutes, and the connection re-enters on its own.
- **A provider 429 never counts.** It moves the request to the next connection or route target. If none can serve, the response is `429 provider_rate_limited` with your provider's `Retry-After`, capped at 300 seconds, rather than a 502 that would invite immediate retries ([errors](https://infro.io/docs/api/errors)).
- **Invalid requests and moderation refusals do not count** toward a connection's health either.
- **Ejection is visible.** The console shows each connection's health and cool-off deadline, a `connection.failing` [alert](https://infro.io/docs/observability/alerts) is raised as critical while a connection is out of rotation, and a request for a model whose only connection is ejected gets `503` with code `connection_out_of_scope` instead of a message telling you to connect a provider you already connected.

Two limits of the current behaviour are worth designing around. INFRO does not hold a throttled connection back for the provider's `Retry-After`: the next request tries it again in order, so a throttled primary costs one quick failed attempt per request before the next connection serves. And a [route](https://infro.io/docs/routes)'s same-target retries run immediately, without backoff, so on a route whose primary is often throttled, set retries to 0 and let failover do the work.

## What to watch

Every state change of every breaker, throttle gate and parked account should be an event you can query. The signals worth alerting on are the share of requests served by a fallback, time spent open per account, and any parked account — a spend cap is a conversation with finance, not an incident to wait out. Watch the throttle gates too: an account that spends a growing share of each hour gated is telling you its quota no longer fits its traffic, which is a capacity request to file with the provider, and a signal the breaker, by design, will never raise.

## Questions

### Should an Anthropic 529 overloaded error open the circuit breaker?

It is an availability signal: Anthropic describes 529 as the API being temporarily overloaded across users. Count it with timeouts and 5xx, but expect it to affect every account on that provider at once, so the useful failover target is a different provider or region, not a second key.

### How many trial requests should a half-open breaker allow?

Few. The point of half-open is to avoid flooding a recovering service. One trial at a time, closing after it succeeds and reopening with a longer cool-off if it fails, is the simplest safe policy; allow a handful in parallel only for high-traffic targets.

## Sources

1. CircuitBreaker — Martin Fowler. https://martinfowler.com/bliki/CircuitBreaker.html (accessed 2026-09-29)
2. Rate limits — Anthropic. https://platform.claude.com/docs/en/api/rate-limits (accessed 2026-09-29)
3. Rate limits — OpenAI. https://developers.openai.com/api/docs/guides/rate-limits (accessed 2026-09-29)
4. Manage Azure OpenAI in Microsoft Foundry Models quota — Microsoft. https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/quota (accessed 2026-09-29)
5. Claude API errors — Anthropic. https://platform.claude.com/docs/en/api/errors (accessed 2026-09-29)
6. Circuit Breaker pattern — Microsoft Azure Architecture Center. https://learn.microsoft.com/en-us/azure/architecture/patterns/circuit-breaker (accessed 2026-09-29)
7. Handling Overload — Site Reliability Engineering — Google. https://sre.google/sre-book/handling-overload/ (accessed 2026-09-29)