September 3, 2026: provider incidents and failover

What the status pages recorded

The day was widely reported as several AI providers going down at once. That is close enough for a headline and not precise enough for a design review. Each provider's own status page records what it saw, with timestamps, components and, in one case, a cause. Here is the day rebuilt from those pages only, in UTC.

Anthropic, two incidents. The first, "Elevated errors for Claude Sonnet 5", was posted at 12:37, a fix was in place at 12:47 and it was resolved at 12:56¹. The second began at 13:26 with elevated errors on Claude Mythos 5.1, Claude Fable 5.1 and Claude Opus 5. At 13:41 Anthropic said it had identified the cause; at 13:50 it published the full list of affected models: Mythos/Fable 5.1, Mythos/Fable 5, Opus 5, Opus 4.8 and Opus 4.6. At 15:25 the only affected models were Opus 4.8 and Opus 5, with the rest back at their baseline error rate. A fix was deployed at 16:06 and the incident was resolved at 16:23, with impact ending at 16:16. Both incidents list claude.ai, the Claude API, Claude Code and Claude Cowork as components². The cause itself was not described.

xAI, one incident across every surface. xAI's status feed records a "Models outage" posted at 13:30 ("Grok is experiencing issues") and resolved at 17:09 ("traffic is healthy again"). The same incident is listed on the global API (api.x.ai), the US API (us.api.x.ai), grok.com, the iOS and Android apps, Grok in X, Grok Build and the Office and Workspace plugins³. No cause was given.

OpenAI, consumer and Codex surfaces. Shortly after midnight, from 00:04 to 00:10, ChatGPT Work Mode had high error rates⁴. The larger incident, "Elevated errors across ChatGPT and Codex", was first posted at 14:43, mitigated by 15:17 and resolved at 16:55. It lists fourteen ChatGPT components and four Codex components, including the Codex API; the OpenAI API Platform is not among them⁵. OpenAI's status pages show times without a zone, and its RSS feed confirms they are UTC⁶. The published write-up gives the cause: a routing configuration update was misinterpreted and the router stopped sending requests to available services; main impact ran from about 7:43 to 8:20 PDT (14:43 to 15:20 UTC); and when routing was restored, returning traffic overloaded some services and slowed recovery⁷.

Google, nothing posted that this article could find. The Google Cloud status summary lists no Vertex AI incident on 3 September; its most recent entries at the time were from 1 September⁸. Google AI Studio's status page renders in the browser and its history could not be read for this article. Reports that the Gemini API had trouble with newly created keys that morning come from press and forum coverage, not from a status page; treat them as unconfirmed.

What the record says, and what it does not

Three things stand out once the headlines are set aside.

First, for an application calling provider APIs, the overlap was narrower than "everyone was down". From 13:30 to 16:16 the Claude API had elevated errors on specific models and xAI's API was in an outage. OpenAI's posted incident was about ChatGPT and Codex; an application calling OpenAI's general API Platform is not listed as affected.

Second, Anthropic's incident was model-specific. Sonnet 5 had its own short incident in the morning and was not on the afternoon list; from 15:25 even the afternoon incident was down to two models. An application whose fallback was "another Claude model on the same account" would have been served for most of the window, provided that fallback had been evaluated for the task.

Third, no provider has published a common cause, and the posted record does not show one. Overlapping incidents can be coincidence, a shared dependency, or a load shift as users and agents move from one failing provider to another. You cannot tell from status pages, and you should not design on the assumption that you know.

What correlated and model-specific incidents mean for failover

Most failover designs are written against one scenario: the primary provider is down, so send everything to the secondary. The record above shows at least three other shapes, and each wants a different response.

• Model-specific errors. One or several models fail while others on the same account serve normally. The cheapest fallback is another model from the same vendor, if it has been evaluated for the task, or the same model from another host.
• Vendor-wide outage on every surface. Everything from that vendor fails, API included. Only another vendor helps, and only if its output has been evaluated for the workload.
• Correlated incidents across vendors. Two of your candidate providers fail in the same hour. A chain of three targets across two vendors can run out. Decide in advance what the application does when every target fails: queue, degrade or show a clear error.

Same model on another host, or another model

There are two kinds of alternate, and they protect against different failures.

The same model on another host, such as Claude through Anthropic's API and through Google Vertex AI, keeps the output distribution you evaluated and requires no prompt changes. It helps when the fault is in one provider's serving stack. It does not help if the fault follows the model everywhere, and the status pages cannot tell you in advance which kind of fault you are in. Google Cloud's dashboard showing no Vertex AI incident that day is not evidence that Claude on Vertex served normally; it only means nothing was posted.

Another model or vendor survives model-specific and vendor-wide incidents, at the price of different output. Tool-call formats, structured-output behaviour, refusal behaviour and cost per request all change. That makes the fallback a product decision to test before an incident, not a list entry you add during one.

Health ejection and timeouts

A gateway that sends every request to a failing target first, waits for the error, then fails over, pays the failure's latency on every request. Health ejection fixes that: after a run of failures, take the target out of rotation for a short, growing window, and let one success put it back. Ejection should be per connection, not per vendor, so one failing key does not take its healthy siblings down with it.

Timeouts need two numbers: a per-attempt timeout that ends a stalled call, and a total budget that bounds how long the user waits across every attempt. A chain of three targets with a 60-second per-attempt timeout is a three-minute wait in the worst case, which for an interactive request is an outage of its own.

Retries, retry storms and idempotency

OpenAI's write-up describes the mechanism that makes incidents longer: when routing came back, returning traffic overloaded services and slowed recovery⁷. Every client retrying immediately and repeatedly is that traffic.

• Retry a failing target only for failures that retrying fixes. A 5xx during a declared incident usually is not one; moving to the next target is.
• On a 429, honour Retry-After for that key and move to another key or account, not back to the same one.
• Cap retries per request and per client, and add jitter where you back off, so recovering services do not see a synchronized wave.
• Never retry a request the provider rejected as malformed (400, 404, 422). Another target would refuse it too.

Idempotency is the quiet part. A completion request has no side effects at the provider beyond billing, but your application's handling of the response often does: sending a message, writing a record, calling a tool. A retried request that succeeded the first time, behind a timeout, can do that twice. Make the side effects idempotent with a request ID you generate, and do not restart a streamed response that has already reached the user.

The attempt trail

After the incident someone will ask what happened to requests between 13:30 and 16:16. The useful answer is per request: which targets were tried, in what order, what each one returned, how long each took, and which one finally served. Without that trail, "failover worked" is an assertion. With it, you can count how many requests failed over, how much latency failover added, and whether any request exhausted every target.

Drills

A fallback that has never served traffic is a guess. Drills turn it into a known path: disable a target in staging, inject 5xx and 429 responses, and confirm the chain behaves as written, including what the application shows when every target fails. The failover drills guide has a schedule and scenarios.

Failover and routing rules in INFRO

INFRO routes requests over provider connections you own, so every alternate below is an account and contract you already hold. This section describes the behaviour in the routes and providers docs; the routing rules guide covers configuration in more depth.

Routes and targets. A route is a name your application sends as model, backed by up to 10 targets across your connections, walked in order (priority) or picked by weight (weighted) and then walked in descending weight. A target that errors or times out is abandoned and the next one runs, before any bytes reach your client. A target whose connection is disabled, ejected or outside the key's environment is skipped. A route can set a per-attempt timeout from 1 second to 10 minutes and up to 3 retries on the same target; those retries run immediately, without backoff, which is a reason to choose the failure rules below carefully.

Failure rules. A route's failover object sets an action per failure kind: rate_limit (429), server_error (5xx), timeout, connection (DNS, TLS or connection failure), credential (401/403) and moderation (the provider's safety system refused). Each takes retry (the default: the route's retries on the same target, then the next target), failover (the next target at once) or stop (return the error now). For an incident like Anthropic's, server_error: failover moves to the next target without retrying a model that is failing. A rejected request (400, 404, 422) always ends the request.

Connection health. Each connection keeps its own failure count. Consecutive failures that blame the provider eject the connection for a bounded window that grows if failures continue, and one success clears it; a failing production key does not eject a healthy staging key of the same provider. The console shows each connection's health, its consecutive failures, the ejection deadline and the last error, including the provider's own message with credentials redacted.

Alternate hosts and key spreading. A connection can list up to five more API hosts for the same key, tried in order on a DNS, TLS or connection failure, a timeout, or an edge 502, 503 or 504. A host that failed is skipped for a minute, and switching host spends no route retry. A provider's own 500 or 429 does not switch hosts. Requests that name a model directly, without a route, are spread across a provider's active keys, with keys marked standby tried only after every active key failed; a key that just answered 429 goes to the back of the line for the provider's Retry-After, at most five minutes.

Claude alternates. INFRO's Google Vertex AI kind serves Claude models on your Google Cloud project alongside Gemini, so a route can list Claude through your Anthropic connection first and the same model through your Vertex connection second; check the providers catalog for which Claude versions each kind carries. INFRO's AWS Bedrock kind uses Bedrock's OpenAI-compatible endpoint, which serves OpenAI, xAI, Moonshot and MiniMax models and not Claude, so it is not a Claude alternate in INFRO today.

Streams and the attempt trail. Once a stream has started, a failure ends that stream; INFRO does not restart a response the client has already begun receiving. Every attempt appears on the request's trace with the connection that made it, the host when the connection has several, and the error that ended it, and the request CSV export carries a fallback_attempts count per request, so the question "what happened to requests during the incident" has a per-request answer.

What to change this quarter

Read your providers' status pages for the incidents of the last quarter and classify them: model-specific, vendor-wide, or control-plane only. Check that your fallback chain has an answer for each, and that every fallback model has been evaluated for the workload it would serve. Set failure rules so that a declared 5xx moves on instead of retrying. Put a total time budget on interactive requests. Make sure your request records keep the attempt trail. Then run a drill, because 3 September will not be the last day like it. The automatic failover guide walks through the full design.