Circuit breakers for LLM providers: throttling is not failure

A circuit breaker has one job: stop sending requests to something that is failing, so callers fail fast instead of waiting, and the failing thing gets room to recover. Fowler's description is the one most implementations follow — a closed breaker passes calls and counts failures, an open one returns an error immediately, and after a timeout a half-open breaker lets a trial call through to see whether the fault has cleared¹.

The pattern assumes an error means the dependency is unwell. For LLM providers that assumption breaks on the most common error of all.

The 429 problem

A 429 from a model provider usually means the opposite of unwell. The account answered, promptly, and said: this account has used its allowance for now. Anthropic's limits refill continuously as a token bucket²; OpenAI's are windows per organization, project and model³. In both cases, capacity returns on its own within seconds to a minute, and nothing is broken.

Count those 429s as failures and three things go wrong.

• A busy account is ejected for being busy. The breaker opens exactly when traffic is highest, removing capacity at peak.
• The ejection outlasts the throttle. A rate window reopens in seconds; a breaker cool-off is typically tens of seconds to minutes. The breaker turns a short slowdown into a longer outage of that account.
• It cascades. Traffic moves to the next account, which is now over its limit too, whose breaker opens, and so on. With every account open, the application fails requests that the providers would have served a few seconds later.

Hammering is no better than ejecting. OpenAI and Azure both state that unsuccessful requests count toward the per-minute limit³⁴, so a client that keeps retrying a throttled account keeps it throttled.

Three conditions, three mechanisms

The fix is to stop asking one mechanism to handle three different conditions.

1. Availability failure — timeouts, connection errors, 5xx, and overload errors such as Anthropic's 529 overloaded_error⁵. This is what the breaker is for. Count it, open on a threshold, cool off, probe.
2. Throttling — a 429 with a wait hint. Handle it with a throttle gate: mark the account unavailable until the time the provider named, route around it meanwhile, and let it back without a probe when the time passes. Azure's guidance calls this accelerated circuit breaking: when a failure response itself says how long to wait, the breaker can trip immediately for that long⁶.
3. Quota or spend exhaustion — a 429 or 400 that names a cap. Anthropic's tier spend cap, for example, returns 429 with no retry-after and keeps failing until access resumes². Park the account until the period resets and tell a person; neither a probe nor a timer will fix it.

Credential rejections belong with availability, but deserve a lower threshold and a page: an account whose key was revoked will not recover by waiting. Invalid requests — a 400 for a malformed body — say nothing about the account at all.

Scope each breaker to one quota and one failure domain

Microsoft's pattern notes warn against a single breaker for a resource that has several independent providers behind it: one healthy part gets blocked while a failing one keeps receiving traffic⁶. For LLM traffic the independent parts are provider accounts, deployments and regions, and often models, since rate limits are applied per model³².

• Key the throttle gate by the quota scope: the account and model, or the Azure deployment.
• Key the availability breaker by the failure domain: the account, deployment or region endpoint.
• Key quota parking by whatever the cap applies to — usually the whole account or project.

Not every overload is local. Anthropic describes 529 as API-wide load and separately documents acceleration limits that return 429 when an organization's usage rises sharply⁵. Azure documents capacity throttling and temporary rate-limit reductions on shared deployments, and warns that these are often misread as quota problems⁴. When every account at one provider trips together, the useful failover target is a different provider or region, not a second key.

A breaker that tells the difference

The class below keeps the three mechanisms separate. It opens on a failure rate over a rolling window, lengthens the open period each time it reopens, allows one trial at a time when half-open, and logs every state change, which Fowler singles out as the most useful signal a breaker produces¹.

export type Verdict = "success" | "availability" | "credential" | "throttled" | "quota" | "request";

const QUOTA_CODES = new Set([
  "enforced_spend_limit_reached",       // Anthropic tier spend cap
  "organization_spend_limit_exceeded",  // OpenAI
  "project_spend_limit_exceeded",       // OpenAI
]);

export function classify(status: number | null, body: unknown): Verdict {
  if (status === null) return "availability";                // connection error or timeout
  if (status < 400) return "success";
  const err = (body as { error?: { code?: string; details?: { error_code?: string } } })?.error;
  const code = err?.code ?? err?.details?.error_code ?? null;
  if (code && QUOTA_CODES.has(code)) return "quota";
  if (status === 429) return "throttled";
  if (status === 401) return "credential";
  if (status === 403) return "credential";                   // unless you can identify a moderation refusal
  if (status === 408 || status >= 500) return "availability"; // includes 529
  return "request";                                          // 400, 404, 422: not the account's fault
}

type State = "closed" | "open" | "half_open";

export class ProviderBreaker {
  private state: State = "closed";
  private window: boolean[] = [];        // true = availability failure
  private credentialFailures = 0;
  private opens = 0;
  private openUntil = 0;
  private probeInFlight = false;
  private throttledUntil = 0;
  private parkedUntil = 0;

  constructor(
    readonly name: string,
    private readonly o = {
      windowSize: 20, minCalls: 10, failureRate: 0.5, credentialTrips: 2,
      baseOpenMs: 30_000, maxOpenMs: 15 * 60_000, defaultThrottleMs: 5_000,
    },
    private readonly log: (msg: string) => void = console.log,
  ) {}

  /** May a request go to this account now? Call immediately before sending. */
  canSend(now = Date.now()): boolean {
    if (now < this.parkedUntil || now < this.throttledUntil) return false;
    if (this.state === "open") {
      if (now < this.openUntil) return false;
      this.transition("half_open");
    }
    if (this.state === "half_open") {
      if (this.probeInFlight) return false;
      this.probeInFlight = true;                  // exactly one trial at a time
    }
    return true;
  }

  record(v: Verdict, hint: { retryAfterMs?: number | null; resetsAt?: number } = {}, now = Date.now()) {
    this.probeInFlight = false;
    switch (v) {
      case "success":
        return this.healthy();
      case "throttled":                           // healthy and busy: gate, do not count
        this.throttledUntil = now + (hint.retryAfterMs ?? this.o.defaultThrottleMs);
        return this.healthy();
      case "quota":
        this.parkedUntil = hint.resetsAt ?? now + 60 * 60_000;
        this.log(`${this.name}: parked until ${new Date(this.parkedUntil).toISOString()} (quota or spend cap)`);
        return;
      case "request":
        return;                                   // says nothing about the account
      case "credential":
        this.credentialFailures += 1;
        if (this.credentialFailures >= this.o.credentialTrips) return this.trip(now);
        return this.fail(now);
      case "availability":
        return this.fail(now);
    }
  }

  private healthy() {
    this.credentialFailures = 0;
    this.push(false);
    if (this.state === "half_open") {
      this.opens = 0;
      this.window = [];
      this.transition("closed");
    }
  }

  private fail(now: number) {
    if (this.state === "half_open") return this.trip(now);   // failed trial: reopen, longer
    this.push(true);
    const failures = this.window.filter(Boolean).length;
    if (this.window.length >= this.o.minCalls && failures / this.window.length >= this.o.failureRate) {
      this.trip(now);
    }
  }

  private trip(now: number) {
    this.opens += 1;
    const ms = Math.min(this.o.maxOpenMs, this.o.baseOpenMs * 2 ** (this.opens - 1));
    this.openUntil = now + ms;
    this.window = [];
    this.transition("open", `for ${Math.round(ms / 1000)}s`);
  }

  private push(failed: boolean) {
    this.window.push(failed);
    if (this.window.length > this.o.windowSize) this.window.shift();
  }

  private transition(to: State, detail = "") {
    if (this.state === to) return;
    this.log(`${this.name}: ${this.state} -> ${to} ${detail}`.trim());
    this.state = to;
  }
}

A router then takes the first account whose breaker says canSend(), calls it, and passes the verdict back with record(), including the parsed Retry-After for a throttle. When no account can send, fail fast with an error your application handles, rather than queueing blind. And keep retries bounded independently of the breaker: Google's SRE book describes a per-client retry budget that stops retrying once retries exceed about 10% of requests, which caps the extra load a fleet can add during an incident⁷.

Tuning without guessing

Microsoft recommends matching the breaker to the likely recovery pattern: open too long and it rejects requests after the fault has cleared; move to half-open too soon and it flaps⁶. Fowler adds that thresholds can differ by error type, for example tolerating more timeouts than connection failures¹. For provider accounts, a sensible starting point is a short first cool-off that grows while failures continue, a failure-rate threshold with a minimum number of calls so that two early errors cannot trip it, and a much lower threshold for rejected credentials. Then verify it with a drill rather than trusting the numbers.

How INFRO takes a connection out of rotation

INFRO applies this separation to each provider connection you connect, per connection rather than per provider kind, so a staging key failing does not eject a production key (providers).

• Availability failures and credential rejections count. Five in a row eject the connection for availability errors, two for a rejected credential. The cool-off starts at 30 seconds and grows while failures continue, to at most 15 minutes, and the connection re-enters on its own.
• A provider 429 never counts. It moves the request to the next connection or route target. If none can serve, the response is 429 provider_rate_limited with your provider's Retry-After, capped at 300 seconds, rather than a 502 that would invite immediate retries (errors).
• Invalid requests and moderation refusals do not count toward a connection's health either.
• Ejection is visible. The console shows each connection's health and cool-off deadline, a connection.failing alert is raised as critical while a connection is out of rotation, and a request for a model whose only connection is ejected gets 503 with code connection_out_of_scope instead of a message telling you to connect a provider you already connected.

Two limits of the current behaviour are worth designing around. INFRO does not hold a throttled connection back for the provider's Retry-After: the next request tries it again in order, so a throttled primary costs one quick failed attempt per request before the next connection serves. And a route's same-target retries run immediately, without backoff, so on a route whose primary is often throttled, set retries to 0 and let failover do the work.

What to watch

Every state change of every breaker, throttle gate and parked account should be an event you can query. The signals worth alerting on are the share of requests served by a fallback, time spent open per account, and any parked account — a spend cap is a conversation with finance, not an incident to wait out. Watch the throttle gates too: an account that spends a growing share of each hour gated is telling you its quota no longer fits its traffic, which is a capacity request to file with the provider, and a signal the breaker, by design, will never raise.