What is the difference between a retry and a fallback?

A retry sends the same request to the same provider again. A fallback sends it to a different provider account or a different model. Retries help with brief glitches; fallbacks help when a provider stays unavailable.

Should a 429 open a circuit breaker?

No. A rate-limited account is busy, not broken. Count only server errors, timeouts and connection failures toward the breaker, and route a rate-limited request to another account instead.

How many times should I retry an LLM request?

Usually once, or not at all, when a user is waiting. Every attempt has to fit inside the caller's deadline, and the official SDKs already retry twice by default unless you turn that off.

LLM failover patterns: retry, fallback, circuit breaker

The three patterns, and what each is for

Pattern	What it does	Use it for	Never use it for
Retry	Sends the same request to the same provider again, after a wait	A brief network error, a 500, a `slow_down`, a 529 that clears in seconds	Spend or quota errors, 400s, anything once the caller's deadline is near
Fallback	Sends the request to a different provider account or a different model	A provider that stays rate limited, overloaded or down while a user waits	Spend limits (it spends around them), untested substitute models
Circuit breaker	Stops calling a provider that keeps failing, then lets one probe through	Outages longer than one request, so each request stops paying to rediscover them	429 rate limits: a busy provider is not a broken one

The three are layers of one policy, not alternatives to each other.

Retries alone make outages worse: every layer that retries a few times multiplies traffic into a provider that is already struggling, and failed requests still count toward your rate limit at OpenAI. Fallback alone wastes time: every request waits for the primary to fail before it moves. The breaker is what makes fallback fast during a long outage.

Classify the failure before you act on it

The question for every failure is whether waiting, or a different target, could plausibly succeed.

Failure	Retry same target?	Fall back?	Counts toward the breaker?
Connection error, DNS, TLS, timeout before a response	Once, if the deadline allows	Yes	Yes
`500`, `502`, OpenAI `503 server_is_overloaded`	Once, after `retry-after`	Yes	Yes
Anthropic `529 overloaded_error`	Once, briefly	Yes	Yes
`429` rate limit or `slow_down`	After `retry-after`, if the caller can wait	Yes, to another account	No
`429` spend, quota or balance code (OpenAI)	No	No	No
Anthropic tier spend cap (`429`, no `retry-after`)	No	No	No
A spend limit you set on Anthropic (`400`)	No	No	No
`400`, `422` invalid request	No	Usually no	No
`401`, `403`	No	Yes, and alert	Yes: the key is broken

From OpenAI's error-codes guide and Anthropic's errors and rate-limit pages. Check each provider's own documentation for the account you connect.

Two rows trip people up. A 429 is not always a rate limit: OpenAI uses it for six different errors, four of which are about money (the 429 guide lists them). And Anthropic signals a spend limit you set with a 400, so "400 means a malformed request" is not safe either.

Retry: few attempts, jittered, inside a deadline

Both official OpenAI SDKs and Anthropic's SDKs retry twice by default. OpenAI's retry every 429, whatever its code. That default stacks with any retry loop you write and any gateway you put in front, so decide where retries live and turn the others off.

Three rules keep retries honest:

Read the error before you retry, and honour retry-after. Otherwise use exponential backoff with jitter, so a thousand clients do not retry in the same instant.
Fit every attempt inside the caller's deadline. If a chat request gives up after 30 seconds, a 20-second per-attempt timeout leaves the fallback 10 seconds; a 12-second timeout leaves it 18. Choose the per-attempt timeout from the deadline, not the other way round.
Budget retries: cap them as a share of normal traffic, so a widespread failure cannot turn into a retry storm.

import OpenAI, { APIError } from "openai";

// Turn the SDK's own retries off: it retries every 429, including the ones
// that cannot succeed. The loop below reads the error code first.
const client = new OpenAI({ maxRetries: 0, timeout: 30_000 });

// 429s that no amount of waiting fixes: billing, spend limits and quota.
const QUOTA_CODES = new Set([
  "credit_balance_exhausted",
  "organization_spend_limit_exceeded",
  "project_spend_limit_exceeded",
  "organization_usage_limit_exceeded",
  "insufficient_quota",
]);

export function isRetryable(err: unknown): err is APIError {
  if (!(err instanceof APIError)) return false;
  if (err.status === undefined) return true; // connection error or timeout
  if (err.status === 429) return !QUOTA_CODES.has(String(err.code)) && !QUOTA_CODES.has(String(err.type));
  return err.status === 408 || err.status === 409 || err.status >= 500;
}

// Wait what the server asks for; otherwise exponential backoff with jitter.
export function retryDelayMs(err: APIError, attempt: number): number {
  const header = (name: string) => Number(err.headers?.get(name));
  if (header("retry-after-ms") > 0) return header("retry-after-ms");
  if (header("retry-after") > 0) return header("retry-after") * 1000;
  const ceiling = Math.min(500 * 2 ** attempt, 8_000);
  return ceiling / 2 + Math.random() * (ceiling / 2);
}

export async function withRetries<T>(
  call: () => Promise<T>,
  { maxAttempts = 5, maxWaitMs = 20_000 } = {},
): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await call();
    } catch (err) {
      if (!isRetryable(err) || attempt + 1 >= maxAttempts) throw err;
      const wait = retryDelayMs(err, attempt);
      // A user is waiting: past this, fail over or fail fast instead of sleeping.
      if (wait > maxWaitMs) throw err;
      await new Promise((resolve) => setTimeout(resolve, wait));
    }
  }
}

try {
  const completion = await withRetries(() =>
    client.chat.completions.create({
      model: "gpt-5.4-mini",
      messages: [{ role: "user", content: "Summarize this ticket in one line." }],
    }),
  );
  console.log(completion.choices[0]?.message.content);
} catch (err) {
  if (err instanceof APIError) {
    console.error(`gave up: ${err.status} ${err.code ?? err.type} (request ${err.requestID})`);
    process.exitCode = 1;
  } else {
    throw err;
  }
}

import random
import sys
import time

import openai
from openai import OpenAI

# Turn the SDK's own retries off: it retries every 429, including the ones
# that cannot succeed. The loop below reads the error code first.
client = OpenAI(max_retries=0, timeout=30.0)

# 429s that no amount of waiting fixes: billing, spend limits and quota.
QUOTA_CODES = {
    "credit_balance_exhausted",
    "organization_spend_limit_exceeded",
    "project_spend_limit_exceeded",
    "organization_usage_limit_exceeded",
    "insufficient_quota",
}


def is_retryable(err: Exception) -> bool:
    if isinstance(err, openai.APIConnectionError):  # includes timeouts
        return True
    if not isinstance(err, openai.APIStatusError):
        return False
    if err.status_code == 429:
        return err.code not in QUOTA_CODES and err.type not in QUOTA_CODES
    return err.status_code in (408, 409) or err.status_code >= 500


def retry_delay(err: Exception, attempt: int) -> float:
    """Wait what the server asks for; otherwise exponential backoff with jitter."""
    headers = err.response.headers if isinstance(err, openai.APIStatusError) else {}
    for name, divisor in (("retry-after-ms", 1000), ("retry-after", 1)):
        try:
            value = float(headers.get(name, ""))
        except ValueError:
            continue
        if value > 0:
            return value / divisor
    ceiling = min(0.5 * 2**attempt, 8.0)
    return ceiling / 2 + random.random() * ceiling / 2


def with_retries(call, max_attempts=5, max_wait=20.0):
    for attempt in range(max_attempts):
        try:
            return call()
        except Exception as err:
            if not is_retryable(err) or attempt + 1 >= max_attempts:
                raise
            wait = retry_delay(err, attempt)
            # A user is waiting: past this, fail over or fail fast instead of sleeping.
            if wait > max_wait:
                raise
            time.sleep(wait)


if __name__ == "__main__":
    try:
        completion = with_retries(
            lambda: client.chat.completions.create(
                model="gpt-5.4-mini",
                messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
            )
        )
        print(completion.choices[0].message.content)
    except openai.APIStatusError as err:
        print(f"gave up: {err.status_code} {err.code or err.type} (request {err.request_id})", file=sys.stderr)
        sys.exit(1)

The same retry loop as the 429 guide. Tested: transient errors are retried, spend errors are not, five 5xx end in a clear error.

Fallback: to a target you have tested for this task

Fallback changes who answers. To another account for the same model, behaviour stays the same. To a different model, output length, formatting, tool-call reliability and refusals all shift, so allow it per feature and only for models that passed the same evaluation as the primary. When nothing tested is available, a clear degraded result ("drafting is temporarily unavailable") beats a silent quality change.

import OpenAI, { APIError } from "openai";
import Anthropic from "@anthropic-ai/sdk";

// No retries on OpenAI: a rate-limited provider stays limited for seconds, so
// the fallback IS the retry. Both clients read their keys from the environment.
const openai = new OpenAI({ maxRetries: 0, timeout: 20_000 });
const anthropic = new Anthropic({ maxRetries: 1, timeout: 20_000 });

// Never fail over on these: your own spend or quota limit said stop, and
// moving the traffic to another provider would spend around it.
const QUOTA_CODES = new Set([
  "credit_balance_exhausted",
  "organization_spend_limit_exceeded",
  "project_spend_limit_exceeded",
  "organization_usage_limit_exceeded",
  "insufficient_quota",
]);

function shouldFailOver(err: unknown): boolean {
  if (!(err instanceof APIError)) return false;
  if (err.status === undefined) return true; // connection error or timeout
  if (err.status === 429) return !QUOTA_CODES.has(String(err.code)) && !QUOTA_CODES.has(String(err.type));
  return err.status === 408 || err.status >= 500;
}

export async function answer(prompt: string): Promise<{ text: string; servedBy: string }> {
  try {
    const res = await openai.chat.completions.create({
      model: "gpt-5.4-mini",
      messages: [{ role: "user", content: prompt }],
    });
    return { text: res.choices[0]?.message.content ?? "", servedBy: "openai" };
  } catch (err) {
    if (!shouldFailOver(err)) throw err;
    const res = await anthropic.messages.create({
      model: "claude-haiku-5-5",
      max_tokens: 1024,
      messages: [{ role: "user", content: prompt }],
    });
    const text = res.content.map((block) => (block.type === "text" ? block.text : "")).join("");
    return { text, servedBy: "anthropic" };
  }
}

const { text, servedBy } = await answer("Draft a two-line status update for a delayed order.");
// Log who served it: a fallback that nobody can see becomes a silent quality change.
console.log(`[${servedBy}] ${text}`);

import anthropic
import openai

# No retries on OpenAI: a rate-limited provider stays limited for seconds, so
# the fallback IS the retry. Both clients read their keys from the environment.
openai_client = openai.OpenAI(max_retries=0, timeout=20.0)
anthropic_client = anthropic.Anthropic(max_retries=1, timeout=20.0)

# Never fail over on these: your own spend or quota limit said stop, and
# moving the traffic to another provider would spend around it.
QUOTA_CODES = {
    "credit_balance_exhausted",
    "organization_spend_limit_exceeded",
    "project_spend_limit_exceeded",
    "organization_usage_limit_exceeded",
    "insufficient_quota",
}


def should_fail_over(err: Exception) -> bool:
    if isinstance(err, openai.APIConnectionError):  # includes timeouts
        return True
    if not isinstance(err, openai.APIStatusError):
        return False
    if err.status_code == 429:
        return err.code not in QUOTA_CODES and err.type not in QUOTA_CODES
    return err.status_code == 408 or err.status_code >= 500


def answer(prompt: str) -> tuple[str, str]:
    try:
        res = openai_client.chat.completions.create(
            model="gpt-5.4-mini",
            messages=[{"role": "user", "content": prompt}],
        )
        return res.choices[0].message.content or "", "openai"
    except Exception as err:
        if not should_fail_over(err):
            raise
        res = anthropic_client.messages.create(
            model="claude-haiku-5-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        text = "".join(block.text for block in res.content if block.type == "text")
        return text, "anthropic"


if __name__ == "__main__":
    text, served_by = answer("Draft a two-line status update for a delayed order.")
    # Log who served it: a fallback that nobody can see becomes a silent quality change.
    print(f"[{served_by}] {text}")

Tested: a 429 moves the request to Anthropic after one OpenAI attempt; a spend-limit 429 does not.

Circuit breaker: stop paying to rediscover an outage

A breaker has three states. Closed: traffic flows and failures are counted in a time window. Open: after enough failures, the provider gets no traffic and requests go straight to the fallback. Half-open: after a cool-down, exactly one probe request goes through; success closes the breaker, failure opens it again.

Starting values that work for most chat traffic: open after 3 to 5 failures within 30 seconds, cool down for 30 seconds, and grow the cool-down if the probe keeps failing. Keep one breaker per provider account, not one per model, and never count a 429: a rate-limited account is busy, and ejecting it for being busy moves its load onto everything else.

import OpenAI, { APIError } from "openai";
import Anthropic from "@anthropic-ai/sdk";

type State = "closed" | "open" | "half-open";

/**
 * Stops sending traffic to a provider that keeps failing, then lets one probe
 * through after a cool-down. Counts only failures that say the provider is
 * unhealthy: a 429 means "busy", not "broken", and must not open the breaker.
 */
export class CircuitBreaker {
  state: State = "closed";
  private failures: number[] = [];
  private openedAt = 0;
  private probing = false;
  private readonly threshold: number;
  private readonly windowMs: number;
  private readonly cooldownMs: number;

  constructor(opts: { threshold: number; windowMs: number; cooldownMs: number }) {
    this.threshold = opts.threshold;
    this.windowMs = opts.windowMs;
    this.cooldownMs = opts.cooldownMs;
  }

  allowRequest(now = Date.now()): boolean {
    if (this.state === "open" && now - this.openedAt >= this.cooldownMs) {
      this.state = "half-open";
      this.probing = false;
    }
    if (this.state === "closed") return true;
    if (this.state === "half-open" && !this.probing) {
      this.probing = true; // exactly one probe at a time
      return true;
    }
    return false;
  }

  recordSuccess() {
    this.state = "closed";
    this.failures = [];
    this.probing = false;
  }

  recordFailure(now = Date.now()) {
    this.failures = this.failures.filter((t) => now - t < this.windowMs);
    this.failures.push(now);
    if (this.state === "half-open" || this.failures.length >= this.threshold) {
      this.state = "open";
      this.openedAt = now;
      this.probing = false;
    }
  }
}

// Unhealthy: server errors, timeouts, broken connections. Not 4xx, not 429.
export function isProviderFailure(err: unknown): boolean {
  if (!(err instanceof APIError)) return false;
  return err.status === undefined || err.status >= 500;
}

const openai = new OpenAI({ maxRetries: 0, timeout: 20_000 });
const anthropic = new Anthropic({ maxRetries: 1, timeout: 20_000 });
// Production values are seconds to minutes; 100 ms keeps this demo fast.
const openaiBreaker = new CircuitBreaker({ threshold: 3, windowMs: 30_000, cooldownMs: 100 });

export async function answer(prompt: string): Promise<string> {
  if (openaiBreaker.allowRequest()) {
    try {
      const res = await openai.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [{ role: "user", content: prompt }],
      });
      openaiBreaker.recordSuccess();
      return `[openai] ${res.choices[0]?.message.content ?? ""}`;
    } catch (err) {
      if (isProviderFailure(err)) openaiBreaker.recordFailure();
      else if (!(err instanceof APIError && err.status === 429)) throw err;
    }
  }
  const res = await anthropic.messages.create({
    model: "claude-haiku-5-5",
    max_tokens: 1024,
    messages: [{ role: "user", content: prompt }],
  });
  return `[anthropic] ${res.content.map((b) => (b.type === "text" ? b.text : "")).join("")}`;
}

for (let i = 1; i <= 5; i++) {
  console.log(`request ${i}: ${await answer("Tag this ticket.")} (breaker ${openaiBreaker.state})`);
}
await new Promise((r) => setTimeout(r, 150)); // the cool-down passes
console.log(`request 6: ${await answer("Tag this ticket.")} (breaker ${openaiBreaker.state})`);

import threading
import time

import anthropic
import openai


class CircuitBreaker:
    """Stops sending traffic to a provider that keeps failing, then lets one
    probe through after a cool-down. Counts only failures that say the provider
    is unhealthy: a 429 means "busy", not "broken", and must not open it."""

    def __init__(self, threshold: int, window: float, cooldown: float):
        self.threshold, self.window, self.cooldown = threshold, window, cooldown
        self.state = "closed"
        self.failures: list[float] = []
        self.opened_at = 0.0
        self.probing = False
        self.lock = threading.Lock()

    def allow_request(self) -> bool:
        with self.lock:
            now = time.monotonic()
            if self.state == "open" and now - self.opened_at >= self.cooldown:
                self.state, self.probing = "half-open", False
            if self.state == "closed":
                return True
            if self.state == "half-open" and not self.probing:
                self.probing = True  # exactly one probe at a time
                return True
            return False

    def record_success(self) -> None:
        with self.lock:
            self.state, self.failures, self.probing = "closed", [], False

    def record_failure(self) -> None:
        with self.lock:
            now = time.monotonic()
            self.failures = [t for t in self.failures if now - t < self.window] + [now]
            if self.state == "half-open" or len(self.failures) >= self.threshold:
                self.state, self.opened_at, self.probing = "open", now, False


def is_provider_failure(err: Exception) -> bool:
    """Unhealthy: server errors, timeouts, broken connections. Not 4xx, not 429."""
    if isinstance(err, openai.APIConnectionError):
        return True
    return isinstance(err, openai.APIStatusError) and err.status_code >= 500


gpt = openai.OpenAI(max_retries=0, timeout=20.0)
claude = anthropic.Anthropic(max_retries=1, timeout=20.0)
# Production values are seconds to minutes; 0.1 s keeps this demo fast.
openai_breaker = CircuitBreaker(threshold=3, window=30.0, cooldown=0.1)


def answer(prompt: str) -> str:
    if openai_breaker.allow_request():
        try:
            res = gpt.chat.completions.create(
                model="gpt-5.4-mini",
                messages=[{"role": "user", "content": prompt}],
            )
            openai_breaker.record_success()
            return f"[openai] {res.choices[0].message.content}"
        except Exception as err:
            if is_provider_failure(err):
                openai_breaker.record_failure()
            elif not (isinstance(err, openai.APIStatusError) and err.status_code == 429):
                raise
    res = claude.messages.create(
        model="claude-haiku-5-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return "[anthropic] " + "".join(b.text for b in res.content if b.type == "text")


if __name__ == "__main__":
    for i in range(1, 6):
        print(f"request {i}: {answer('Tag this ticket.')} (breaker {openai_breaker.state})")
    time.sleep(0.15)  # the cool-down passes
    print(f"request 6: {answer('Tag this ticket.')} (breaker {openai_breaker.state})")

Tested: three 5xx open the breaker, the next requests skip OpenAI entirely, and one successful probe closes it; five 429s fail over without ever opening it.

The in-memory breaker is right for one process. With many instances, each keeps its own state, which is usually acceptable: every instance discovers the outage after a few requests. Share the state (Redis, or your gateway) only when each discovery is expensive.

Streaming: failover ends at the first token

Before the first token reaches the user, a failed attempt can be dropped and another target tried invisibly. After it, switching targets starts a second, different answer halfway through the first. So the rule is absolute: once a stream has started, a failure ends it with an error the client can detect, and the application decides whether to discard the partial answer and start again. Anthropic can send an overloaded_error mid-stream after a 200; the 529 guide has tested code for exactly that case.

Watch time to first token as well as total duration: a provider that regularly stalls before its first event is failing, even when it eventually answers.

Repeated requests have side effects

Model APIs are POST requests, which HTTP does not treat as safe to repeat. A request that timed out on your side may have completed and been billed on the provider's. Keep side effects (sending a message, writing a record, calling a tool that changes data) out of the model call, behind your own idempotency key, so a late or repeated answer cannot run them twice.

Which pattern, when

Retry once: if the error is transient (connection, 500, 529, slow_down) and the caller's deadline leaves room.
Fall back: if the provider stays rate limited or unavailable and a tested target exists.
Open the breaker: if a provider has failed several times in a short window; skip it until a probe succeeds.
Stop and alert: if the error is a spend, quota or credential problem; no other target should spend around it.
Degrade clearly: if no tested target is available; say what still works.

The INFRO way

The same policy, as configuration in INFRO

INFRO applies these patterns across your own provider connections, before the first token:

Failure rules per kind. On a route, each kind of upstream failure (rate_limit, server_error, timeout, connection, credential, moderation) is set to retry on the same target, failover to the next target at once, or stop. Same-target retries are 0 to 3 per route.
429s are never failures. A provider 429 moves the request on and sends that key to the back of the line for the provider's retry-after (30 seconds when none was sent, at most 5 minutes). It never counts toward ejecting the connection.
A built-in breaker. A connection that fails 5 times in a row (2 for a rejected credential) is taken out of rotation for 30 seconds, growing to at most 15 minutes, and comes back on success.
Budgets and policy are never failed over. A 402 or an allowlist refusal is decided before any provider is called.

Every attempt is on the request's trace, with the connection that made it and the error that ended it.

import OpenAI from "openai";

// The same SDK, pointed at INFRO. Requests run on your own provider accounts.
const client = new OpenAI({
  baseURL: process.env.INFRO_BASE_URL ?? "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
});

const completion = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Summarize this ticket in one line." }],
  // INFRO extension: tried in order once every connection for the model has failed.
  // @ts-expect-error not in the OpenAI SDK's types
  fallbacks: ["anthropic/claude-haiku-5.5"],
});

// `model` is the model that actually served; compare it to detect a fallback.
console.log(completion.model, completion.choices[0]?.message.content);

import os

from openai import OpenAI

# The same SDK, pointed at INFRO. Requests run on your own provider accounts.
client = OpenAI(
    base_url=os.environ.get("INFRO_BASE_URL", "https://api.infro.io/v1"),
    api_key=os.environ["INFRO_API_KEY"],
)

completion = client.chat.completions.create(
    model="openai/gpt-5.4-mini",
    messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
    # INFRO extension: tried in order once every connection for the model has failed.
    extra_body={"fallbacks": ["anthropic/claude-haiku-5.5"]},
)

# `model` is the model that actually served; compare it to detect a fallback.
print(completion.model, completion.choices[0].message.content)

Details in Failover and fallbacks and Routes.

Questions

What is the difference between a retry and a fallback?: A retry sends the same request to the same provider again. A fallback sends it to a different provider account or a different model. Retries help with brief glitches; fallbacks help when a provider stays unavailable.
Should a 429 open a circuit breaker?: No. A rate-limited account is busy, not broken. Count only server errors, timeouts and connection failures toward the breaker, and route a rate-limited request to another account instead.
How many times should I retry an LLM request?: Usually once, or not at all, when a user is waiting. Every attempt has to fit inside the caller's deadline, and the official SDKs already retry twice by default unless you turn that off.

The team building INFRO, a BYOK-first control plane for production AI. Articles are written from the engineering work on the gateway and cite their external sources.

LLM failover patterns: retry vs fallback vs circuit breaker