# Automatic LLM failover: a production implementation guide

URL: https://infro.io/blog/automatic-llm-failover-guide
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-08-28
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Routing and reliability
Format: Guide

> Automatic failover moves a request to another provider account when an attempt fails before any response has reached the client. Do it safely by classifying failures, keeping retries few and jittered, ejecting targets that keep failing, treating the first streamed event as a commit point, and recording every attempt so recovery is visible rather than silent.

Key takeaways:
- Move a request only for failures that another target could plausibly avoid; a malformed request fails everywhere.
- Not every 429 is transient. A throttle clears in seconds; an exhausted spend cap does not.
- Retries amplify load on a struggling system, so keep them few, jittered and inside a total deadline.
- The first streamed event is a commit point. After it, recovery means an error the application handles, not a silent restart.
- Alert on the fallback rate, not just the final error rate. Recovered traffic is the earliest sign of a provider incident.

The first time a provider has a bad afternoon, a system with no failover returns errors for the length of the incident. The second time, after someone has added a retry loop, it often does something worse: it multiplies traffic into the struggling provider, holds user requests open far past their deadline, and succeeds just often enough that nobody notices the bill for the failed attempts.

The incidents of [3 September 2026](https://infro.io/blog/september-3-2026-ai-outages-failover) are a recent example of what a bad afternoon looks like across providers. Good failover is narrower than "retry on error". It moves the right failures, a bounded number of times, only while recovery is still invisible to the user — and it leaves a record of every attempt.

## What failover is, and what it is not

Failover moves a request from a target that failed to a different target that can serve it: another provider account for the same model, or, where explicitly allowed, a different model. It is distinct from two things it is often confused with:

- **A retry** sends the same request to the same target again. It helps with brief glitches and hurts during real incidents.
- **A fallback model** changes what answers the request. It protects availability and changes behaviour, so it is a product decision with an evaluation behind it, not an infrastructure default.

## Classify the failure before you move it

The question for each failure is whether a different target could plausibly succeed. Provider documentation is specific about what each error means, and the details matter.

| Failure | Typical meaning | Move to another target? | Notes |
| --- | --- | --- | --- |
| Connection error, DNS or TLS failure | The target is unreachable | Yes | The request most likely never reached the provider |
| Timeout before any response | The target is slow or stuck | Yes | Only if a total deadline leaves room for another attempt |
| `429` rate or throughput limit | This account's allowance is exhausted for now | Yes, to a different account | OpenAI distinguishes rate limits from sudden traffic increases, both as 429s[1] |
| `429` from a spending cap | The account cannot spend more this period | Yes, but do not retry the same account | Anthropic notes a tier spend-cap 429 carries no `retry-after` header[2] |
| `500`, `502`, `503` | Provider-side failure or overload | Yes | OpenAI documents 503 for temporary model overload, with `Retry-After` when present[1] |
| `529 overloaded_error` | The provider's API is temporarily overloaded | Yes | Documented by Anthropic[2] |
| `400`, `422` invalid request | The request itself is wrong | Usually no | Another provider may validate differently, but a malformed request will usually fail everywhere |
| `400` from a spend limit you set | The account reached a limit its owner configured | Yes, to a different account | Anthropic returns `invalid_request_error` here, not a 429[3], so a 400 is not always a malformed request |
| `401`, `403` | The credential or its permissions are wrong | Yes, and alert | The account needs attention; the request may still succeed elsewhere |

Table: A starting classification. Always check the provider's own error documentation for the account you connect.

When a response carries `Retry-After`, it tells the client how long to wait before sending a follow-up request to that server[4]. For failover that is useful information about the failed target — keep it out of rotation at least that long — rather than a reason to hold the user's request open.

## Retries are not free

Retries add load to a system that is already failing, and when several layers each retry a few times the multiplication is severe[5]. Three practices keep them honest:

1. **Few same-target retries, with jittered backoff.** Adding randomness to backoff spreads retries out instead of synchronizing them into waves[6]. In a request path with a user waiting, zero or one same-target retry is usually right; moving to a different target is the real recovery.
2. **A retry budget.** Cap retries as a proportion of normal traffic per client, so a widespread failure cannot turn into a retry storm. Google's SRE guidance describes a per-client budget where retries stop once they exceed a small fraction of requests[7].
3. **A total deadline, propagated.** Every attempt should fit inside the time the caller is still waiting, and a request that has already missed its deadline should not start another attempt[8].

> **Example: Fitting attempts inside a deadline**
> A chat feature's client gives up after 30 seconds. With a 20-second per-attempt timeout, a stuck primary consumes most of that window and the standby has 10 seconds — often not enough to finish a long answer. With a 12-second per-attempt timeout, a stuck primary still leaves the standby 18 seconds. The right numbers depend on your own time-to-first-token distribution; the point is that the per-attempt timeout is chosen from the deadline, not the other way round.

## Idempotency: what a repeated request can do

HTTP defines which methods are idempotent, and POST — which every model API uses — is not; the specification advises clients not to retry non-idempotent requests automatically unless they know it is safe[9]. For a text generation that matters in two ways. The provider may bill for an attempt that timed out on your side but completed on theirs. And anything your application does with the result — sending a message, calling a tool that writes data — must not run twice because a response arrived late.

Keep side effects out of the model call and behind your own idempotency controls. For asynchronous jobs, an idempotency key lets a client resubmit safely; there is an IETF draft standardizing an `Idempotency-Key` header for exactly this purpose, though it has not become an RFC[10].

## Stop sending traffic to a target that keeps failing

A circuit breaker stops calling a dependency after failures reach a threshold, fails fast while open, and periodically lets a trial request through to see whether it has recovered[11]. For LLM traffic, the equivalent is taking a provider account out of rotation after repeated failures and restoring it after a cool-off or a successful probe. Without it, every request pays the latency of discovering the same outage.

Two cautions. Count throttling separately from errors, or a busy account gets ejected for being busy. And make ejection visible: an account that has been out of rotation for an hour is an incident, even if every request succeeded.

## Streaming turns failover into a commit decision

Streaming responses use server-sent events: the server sends a sequence of events over one long-lived response[12]. Before the first event reaches the client, a failed attempt can be abandoned and another target tried without the client seeing anything. After it, switching targets would start a second, different answer halfway through the first.

[Diagram: The commit point. Failover is invisible only until the first response event is sent; a later failure has to be surfaced to the application as an error.]

So the rule is simple and absolute: once a response has started streaming, a failure ends the stream with an error the client can detect, and the application decides whether to retry the whole request. Watch time to first token as well as total duration — a target that regularly stalls before its first event is failing, even when it eventually answers.

## When a different model is acceptable

Same-model failover to another account changes infrastructure and keeps behaviour. Cross-model fallback changes behaviour: output length, formatting, tool-call reliability, refusal patterns and cost all shift. Allow it per workload, only for substitutes that passed the same evaluation as the primary, and prefer returning a clear degraded result — "drafting is temporarily unavailable" — over silently answering with a model that has never been tested on the task.

## What operators need to see

A request that recovered should look different from one that did not fail at all. For every request, record:

- the initial target and each failed attempt, with its error type and latency;
- the target that finally served, or the error returned if none did;
- whether the failure happened before or after the response started;
- time to first token and total duration.

Then alert on the **fallback rate** per target, not only on final errors. When one account's share of failed attempts rises, final error rates can stay flat for a long time while latency and cost climb.

## Test it before a provider does

- [ ] Force a timeout on the primary and confirm the standby serves within the client's deadline.
- [ ] Return a 429 from the primary and confirm traffic moves to a different account, not back to the same one.
- [ ] Return a 5xx for long enough to trigger ejection, then confirm the target comes back after recovery.
- [ ] Fail a stream after its first event and confirm the client receives an error, not a restarted answer.
- [ ] Disable every target and confirm the application gets a clear error it handles.
- [ ] Find each injected failure in the request records by id, with every attempt listed.

## How failover works in INFRO

In INFRO, failover happens across the targets of a route or across the connections that serve a model — always your own connections, since INFRO holds no provider accounts to fall back to. When an attempt fails before the response has started — a timeout, a connection error, or an error status from the provider — the next eligible target is tried. Errors INFRO raises itself, such as a model-policy or budget refusal, are returned immediately without contacting another provider.

A route's failure rules decide what each kind of failure does: `rate_limit` (429), `server_error` (5xx), `timeout`, `connection` (DNS, TLS or connection failure), `credential` (401/403) and `moderation` (a provider's safety refusal) each map to `retry` (the route's same-target retries, then the next target), `failover` (the next target at once) or `stop` (return this error now). Kinds you leave out keep `retry`. A provider's `400` or `422` on a chat completion is not one of the kinds, and deliberately moves on to the next target, because providers differ on which parameters they accept and the same request can be valid on one host and not another; if every target rejects it, the caller gets a `400` with the provider's own message. That costs an extra attempt of latency on a genuinely malformed request, so validate requests before sending them rather than relying on the gateway to stop one. The full schema is in [routing rules and failure rules](https://infro.io/blog/llm-routing-rules-failure-rules).

Same-target retries are capped at three per route and run without a backoff delay, which is one more reason to keep the number at zero or one. A provider's `429` moves the request on and sends that key to the back of the line for the provider's `Retry-After` (at most five minutes, 30 seconds when none was sent), and never counts toward ejecting the connection. Other failures do: a connection that keeps failing is ejected for a cool-off that starts at 30 seconds and grows to at most 15 minutes, and a single success restores it. A connection can list alternate API hosts for the same key; when its host cannot be reached, the next host is tried as part of the same attempt, spending no retry and counting nothing against the connection. Once a streamed response has started, a failure sends an error event and closes the stream without the final `[DONE]` marker, so a client can tell an interrupted answer from a complete one.

Every attempt is stored with the request. An abbreviated, illustrative record:

```json
{
  "id": "req_…",
  "status": "ok",
  "http_status": 200,
  "model_requested": "support-drafts",
  "model": "anthropic/claude-sonnet-5",
  "route": "standby_a",
  "route_slug": "support-drafts",
  "provider": "aws-bedrock",
  "connection_name": "Bedrock standby",
  "fallback_attempts": [
    {
      "model": "anthropic/claude-sonnet-5",
      "route": "primary",
      "error": "upstream_error",
      "latency_ms": 1840,
      "provider": "anthropic",
      "connection_name": "Anthropic production"
    }
  ],
  "latency_ms": 6120,
  "first_token_ms": 2310
}
```

If every target fails, the error says which failure mattered most: a provider's moderation refusal is returned as such; a request every target rejected as invalid is a `400` with the provider's message; a rate limit anywhere in the chain is a `429` carrying the earliest `Retry-After` a provider sent; `408 request_timeout` when every attempt timed out; otherwise `502 upstream_error`. The last three name each target that was tried and why it failed. See [fallbacks](https://infro.io/docs/features/fallbacks), [routes](https://infro.io/docs/routes) and [errors](https://infro.io/docs/api/errors) for the contract.

## Sources

1. Error codes — OpenAI. https://developers.openai.com/api/docs/guides/error-codes (accessed 2026-10-13)
2. Claude API errors — Anthropic. https://platform.claude.com/docs/en/api/errors (accessed 2026-10-13)
3. Rate limits — Anthropic. https://platform.claude.com/docs/en/api/rate-limits (accessed 2026-10-13)
4. RFC 9110: HTTP Semantics — Retry-After — IETF. https://www.rfc-editor.org/rfc/rfc9110#section-10.2.3 (accessed 2026-10-13)
5. Timeouts, retries, and backoff with jitter — Amazon Builders' Library. https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter (accessed 2026-10-13)
6. Exponential Backoff And Jitter — AWS Architecture Blog. https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ (accessed 2026-10-13)
7. Handling Overload — Site Reliability Engineering — Google. https://sre.google/sre-book/handling-overload/ (accessed 2026-10-13)
8. Addressing Cascading Failures — Site Reliability Engineering — Google. https://sre.google/sre-book/addressing-cascading-failures/ (accessed 2026-10-13)
9. RFC 9110: HTTP Semantics — Idempotent methods — IETF. https://www.rfc-editor.org/rfc/rfc9110#section-9.2.2 (accessed 2026-10-13)
10. The Idempotency-Key HTTP Header Field (Internet-Draft) — IETF HTTPAPI Working Group. https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/ (accessed 2026-10-13)
11. CircuitBreaker — Martin Fowler. https://martinfowler.com/bliki/CircuitBreaker.html (accessed 2026-10-13)
12. HTML Living Standard: Server-sent events — WHATWG. https://html.spec.whatwg.org/multipage/server-sent-events.html (accessed 2026-10-13)

## Revision history

- 2026-10-13: Reviewed against every source and the current gateway. The failure table now notes that Anthropic answers a spend limit you set with a 400, not a 429. INFRO's section is rewritten for what changed since September: failure rules per failure kind on routes, why a provider's 400 still moves to the next target on chat completions, 429s that cool a key without counting toward ejection, alternate hosts, the ejection window, and the full set of errors returned when every target fails.
- 2026-09-11: Expanded from a short overview into a full guide: a failure classification table, retry budgets and deadlines, circuit breaking, the streaming commit point with a diagram, cross-model fallback rules, a failure-injection checklist and how INFRO records attempts. Added cited sources.