The first time a provider has a bad afternoon, a system with no failover returns errors for the length of the incident. The second time, after someone has added a retry loop, it often does something worse: it multiplies traffic into the struggling provider, holds user requests open far past their deadline, and succeeds just often enough that nobody notices the bill for the failed attempts.
The incidents of 3 September 2026 are a recent example of what a bad afternoon looks like across providers. Good failover is narrower than "retry on error". It moves the right failures, a bounded number of times, only while recovery is still invisible to the user — and it leaves a record of every attempt.
What failover is, and what it is not
Failover moves a request from a target that failed to a different target that can serve it: another provider account for the same model, or, where explicitly allowed, a different model. It is distinct from two things it is often confused with:
- A retry sends the same request to the same target again. It helps with brief glitches and hurts during real incidents.
- A fallback model changes what answers the request. It protects availability and changes behaviour, so it is a product decision with an evaluation behind it, not an infrastructure default.
Classify the failure before you move it
The question for each failure is whether a different target could plausibly succeed. Provider documentation is specific about what each error means, and the details matter.
| Failure | Typical meaning | Move to another target? | Notes |
|---|---|---|---|
| Connection error, DNS or TLS failure | The target is unreachable | Yes | The request most likely never reached the provider |
| Timeout before any response | The target is slow or stuck | Yes | Only if a total deadline leaves room for another attempt |
429 rate or throughput limit | This account's allowance is exhausted for now | Yes, to a different account | OpenAI distinguishes rate limits from sudden traffic increases, both as 429s1 |
429 from a spending cap | The account cannot spend more this period | Yes, but do not retry the same account | Anthropic notes a tier spend-cap 429 carries no retry-after header2 |
500, 502, 503 | Provider-side failure or overload | Yes | OpenAI documents 503 for temporary model overload, with Retry-After when present1 |
529 overloaded_error | The provider's API is temporarily overloaded | Yes | Documented by Anthropic2 |
400, 422 invalid request | The request itself is wrong | Usually no | Another provider may validate differently, but a malformed request will usually fail everywhere |
400 from a spend limit you set | The account reached a limit its owner configured | Yes, to a different account | Anthropic returns invalid_request_error here, not a 4293, so a 400 is not always a malformed request |
401, 403 | The credential or its permissions are wrong | Yes, and alert | The account needs attention; the request may still succeed elsewhere |
When a response carries Retry-After, it tells the client how long to wait before sending a follow-up request to that server4. For failover that is useful information about the failed target — keep it out of rotation at least that long — rather than a reason to hold the user's request open.
Retries are not free
Retries add load to a system that is already failing, and when several layers each retry a few times the multiplication is severe5. Three practices keep them honest:
- Few same-target retries, with jittered backoff. Adding randomness to backoff spreads retries out instead of synchronizing them into waves6. In a request path with a user waiting, zero or one same-target retry is usually right; moving to a different target is the real recovery.
- A retry budget. Cap retries as a proportion of normal traffic per client, so a widespread failure cannot turn into a retry storm. Google's SRE guidance describes a per-client budget where retries stop once they exceed a small fraction of requests7.
- A total deadline, propagated. Every attempt should fit inside the time the caller is still waiting, and a request that has already missed its deadline should not start another attempt8.
Idempotency: what a repeated request can do
HTTP defines which methods are idempotent, and POST — which every model API uses — is not; the specification advises clients not to retry non-idempotent requests automatically unless they know it is safe9. For a text generation that matters in two ways. The provider may bill for an attempt that timed out on your side but completed on theirs. And anything your application does with the result — sending a message, calling a tool that writes data — must not run twice because a response arrived late.
Keep side effects out of the model call and behind your own idempotency controls. For asynchronous jobs, an idempotency key lets a client resubmit safely; there is an IETF draft standardizing an Idempotency-Key header for exactly this purpose, though it has not become an RFC10.
Stop sending traffic to a target that keeps failing
A circuit breaker stops calling a dependency after failures reach a threshold, fails fast while open, and periodically lets a trial request through to see whether it has recovered11. For LLM traffic, the equivalent is taking a provider account out of rotation after repeated failures and restoring it after a cool-off or a successful probe. Without it, every request pays the latency of discovering the same outage.
Two cautions. Count throttling separately from errors, or a busy account gets ejected for being busy. And make ejection visible: an account that has been out of rotation for an hour is an incident, even if every request succeeded.
Streaming turns failover into a commit decision
Streaming responses use server-sent events: the server sends a sequence of events over one long-lived response12. Before the first event reaches the client, a failed attempt can be abandoned and another target tried without the client seeing anything. After it, switching targets would start a second, different answer halfway through the first.
Attempt 1 · Target A
- connect and send
- 503 before any response bytes
- recorded as a failed attempt
Attempt 2 · Target B
- response headers
- first response event = commit point
- tokens stream to the client
After commit
So the rule is simple and absolute: once a response has started streaming, a failure ends the stream with an error the client can detect, and the application decides whether to retry the whole request. Watch time to first token as well as total duration — a target that regularly stalls before its first event is failing, even when it eventually answers.
When a different model is acceptable
Same-model failover to another account changes infrastructure and keeps behaviour. Cross-model fallback changes behaviour: output length, formatting, tool-call reliability, refusal patterns and cost all shift. Allow it per workload, only for substitutes that passed the same evaluation as the primary, and prefer returning a clear degraded result — "drafting is temporarily unavailable" — over silently answering with a model that has never been tested on the task.
What operators need to see
A request that recovered should look different from one that did not fail at all. For every request, record:
- the initial target and each failed attempt, with its error type and latency;
- the target that finally served, or the error returned if none did;
- whether the failure happened before or after the response started;
- time to first token and total duration.
Then alert on the fallback rate per target, not only on final errors. When one account's share of failed attempts rises, final error rates can stay flat for a long time while latency and cost climb.
Test it before a provider does
- Force a timeout on the primary and confirm the standby serves within the client's deadline.
- Return a 429 from the primary and confirm traffic moves to a different account, not back to the same one.
- Return a 5xx for long enough to trigger ejection, then confirm the target comes back after recovery.
- Fail a stream after its first event and confirm the client receives an error, not a restarted answer.
- Disable every target and confirm the application gets a clear error it handles.
- Find each injected failure in the request records by id, with every attempt listed.
How failover works in INFRO
In INFRO, failover happens across the targets of a route or across the connections that serve a model — always your own connections, since INFRO holds no provider accounts to fall back to. When an attempt fails before the response has started — a timeout, a connection error, or an error status from the provider — the next eligible target is tried. Errors INFRO raises itself, such as a model-policy or budget refusal, are returned immediately without contacting another provider.
A route's failure rules decide what each kind of failure does: rate_limit (429), server_error (5xx), timeout, connection (DNS, TLS or connection failure), credential (401/403) and moderation (a provider's safety refusal) each map to retry (the route's same-target retries, then the next target), failover (the next target at once) or stop (return this error now). Kinds you leave out keep retry. A provider's 400 or 422 on a chat completion is not one of the kinds, and deliberately moves on to the next target, because providers differ on which parameters they accept and the same request can be valid on one host and not another; if every target rejects it, the caller gets a 400 with the provider's own message. That costs an extra attempt of latency on a genuinely malformed request, so validate requests before sending them rather than relying on the gateway to stop one. The full schema is in routing rules and failure rules.
Same-target retries are capped at three per route and run without a backoff delay, which is one more reason to keep the number at zero or one. A provider's 429 moves the request on and sends that key to the back of the line for the provider's Retry-After (at most five minutes, 30 seconds when none was sent), and never counts toward ejecting the connection. Other failures do: a connection that keeps failing is ejected for a cool-off that starts at 30 seconds and grows to at most 15 minutes, and a single success restores it. A connection can list alternate API hosts for the same key; when its host cannot be reached, the next host is tried as part of the same attempt, spending no retry and counting nothing against the connection. Once a streamed response has started, a failure sends an error event and closes the stream without the final [DONE] marker, so a client can tell an interrupted answer from a complete one.
Every attempt is stored with the request. An abbreviated, illustrative record:
{
"id": "req_…",
"status": "ok",
"http_status": 200,
"model_requested": "support-drafts",
"model": "anthropic/claude-sonnet-5",
"route": "standby_a",
"route_slug": "support-drafts",
"provider": "aws-bedrock",
"connection_name": "Bedrock standby",
"fallback_attempts": [
{
"model": "anthropic/claude-sonnet-5",
"route": "primary",
"error": "upstream_error",
"latency_ms": 1840,
"provider": "anthropic",
"connection_name": "Anthropic production"
}
],
"latency_ms": 6120,
"first_token_ms": 2310
}If every target fails, the error says which failure mattered most: a provider's moderation refusal is returned as such; a request every target rejected as invalid is a 400 with the provider's message; a rate limit anywhere in the chain is a 429 carrying the earliest Retry-After a provider sent; 408 request_timeout when every attempt timed out; otherwise 502 upstream_error. The last three name each target that was tried and why it failed. See fallbacks, routes and errors for the contract.