LLM routing rules: if-rules, failure rules, key spreading

On 3 September 2026, Anthropic's status page recorded elevated errors on several Claude models from 13:26 UTC, narrowing over the afternoon to Opus 4.8 and Opus 5, with impact ending at 16:16 UTC; a shorter incident on Claude Sonnet 5 had run from 12:37 to 12:56¹. The same day, OpenAI's status page recorded elevated errors across ChatGPT and Codex². Incidents like these are rarely "the provider is down". They are usually some models, on some endpoints, for some hours — which is exactly the shape a routing policy can respond to, if the policy exists before the incident does.

This guide is about writing that policy down: the conditions that pick targets, what each kind of failure should do, how to spread traffic across several keys and hosts, and what to record. It builds on LLM routing strategies, which covers priority, weighted and health-aware ordering, and on Automatic LLM failover, which covers commit points and idempotency. The last section shows INFRO's route schema with JSON you can send.

Routing as data, not as a retry loop

Most routing starts as a loop in application code: try provider A, catch, try provider B. It works until someone needs to change it during an incident, or asks why a request took 40 seconds, or discovers that the loop retried a malformed request across four providers. A declarative policy moves those decisions into configuration that can be read, reviewed, versioned and changed without a deploy.

A useful policy answers three questions, in order:

Keeping them separate prevents a common class of bug. An allowlist that a routing rule can widen is not an allowlist. A per-request hint that can add a provider the key was never allowed to reach is a privilege escalation. Eligibility filters should only ever remove candidates, and everything after them should operate on what is left.

Match conditions: what a rule may read

An if-rule is a set of conditions and a target list: when the conditions hold, those targets serve the request instead of the default list. The useful conditions are facts about the request that the caller declares or the gateway knows:

• Who and what for. Tenant tier from request metadata, the product feature, workflow or step, the end user, the API key, the project, the environment.
• Shape. Estimated prompt size, max_tokens, whether it streams, whether it carries images or declares tools.
• When. Hour of day and weekday, for batch windows or regional business hours.

What a rule should not read is the prompt text. Rules that pattern-match on content turn routing configuration into a place where customer content is inspected and logged, and they break silently when someone rewords a system prompt. If you need content-aware routing — small prompts to a small model, hard ones to a large one — make it an explicit classification step whose output is a label, and let rules match the label. The quality-aware routing guide covers that design.

Three semantics are worth deciding deliberately, because they are where rule sets go wrong:

1. First match wins. Evaluate rules top to bottom and stop at the first that holds. Overlapping rules are then resolved by order, which a reviewer can see, rather than by specificity scores nobody can predict.
2. "Not set" is its own value. A request that sends no tier should not match tier != "free". A good evaluator treats a missing value as matching only not_exists, neq and not_in, so a request that says nothing never lands on a rule written for requests that say something specific — or, if your policy wants the opposite, says so explicitly.
3. Fall through, or not. When a matched rule's targets all fail, either continue into the default list or stop. Stopping is right when the rule exists to keep traffic off the default targets — an EU tenant that must stay on EU-hosted connections, say. Falling through is right when the rule merely prefers a target.

Keep rule sets short. Twenty rules nobody can hold in their head are a configuration incident waiting to happen; a dry run that shows which rule a sample request matches, and why, is worth more than another rule.

Classify the failure before you act on it

The single most important property of a failover policy is that it treats different failures differently. Providers make the distinctions explicit. Anthropic documents 429 rate_limit_error for an organization's rate limits or spend caps, 529 overloaded_error for API-wide load, 500 api_error for internal failures, 401 and 403 for credential and permission problems, and 400 invalid_request_error for problems with the request itself³. OpenAI splits its 429s into rate limits, where the advice is to honour Retry-After, and quota or billing limits, where "retrying won't restore access"⁴. RFC 6585 lets a 429 carry a Retry-After header but does not require it⁵.

Some of these deserve more than a cell:

• 429 is about one key or account, not the provider. Trying the same key again before its window resets only adds to the queue. A different key, project or account has its own window and can carry the request now. A rate limit should never count as a health failure, or a busy-but-healthy account gets ejected. The circuit breaker guide goes into that distinction.
• Quota and spend-cap 429s are a stop for that account. Anthropic notes that a tier spend-cap 429 has no retry-after header "and keeps failing until access resumes"³. Move to another account if you have one; otherwise fail fast with a clear message.
• Timeouts are ambiguous. The provider may still be generating, and billing, the answer you gave up on. Retrying a long generation can mean paying for it twice. For interactive traffic, failing over is usually worth it; for long batch generations, a single attempt with a generous timeout can be the better policy.
• Safety refusals are a governance decision. Each provider applies its own policy, so another provider may accept what one refused. Whether that is acceptable is not a reliability question; decide it per workload, and make the default conservative.

Why not to fail over on client errors

A request the provider rejected as malformed — an unknown parameter, an invalid schema, a context window exceeded — will usually be rejected by the next target too. Failing over then costs latency on every hop, and it multiplies load in exactly the moment a bug has made every request bad. The AWS Builders' Library puts the amplification starkly: five layers each retrying three times can put 243 times the load on the component at the bottom⁶. Its guidance is to retry at one layer only, and not to retry errors caused by the request itself⁶.

There is one real exception, and it is specific to multi-host LLM routing: the same model on different hosts does not accept the same requests. Anthropic's own guide for Claude on Amazon Bedrock lists features Bedrock does not support, including URL sources for images and documents, server-side tools, and the API's server-side fallbacks parameter⁷. A request using one of those is a valid request on the Anthropic API and an invalid one on Bedrock. If your targets span hosts, a 400 can be host-specific, and the policy has to choose between "stop" (fast, honest) and "try the next host" (more likely to succeed, slower when the request is genuinely bad). Either way, return the provider's own error message to the caller, so a bad request is visibly bad.

Spreading load across several keys

Rate limits are scoped to keys, projects, workspaces or accounts, depending on the provider. Several keys of one provider are therefore several windows, and a request that one key cannot serve right now another can. The pattern:

• Spread, do not pin. Share requests across the active keys of a provider rather than sending everything to the first one until it fails.
• Cool a throttled key. After a 429, move that key to the back of the line for its Retry-After, with a bound and a default when the provider sent none.
• Hold a standby. Mark one key as standby so it takes traffic only after every active key has failed. Its window is then fresh when you need it, which is the point.
• Name the keys. A key named "OpenAI batch" or "Anthropic standby" makes the trace readable; "key 3" does not.

Same model, different hosts

Many models are served by more than one host: Claude through Anthropic's API, Amazon Bedrock and Google Cloud's Vertex AI; OpenAI models through OpenAI and Azure. Each host is a separate failure domain with its own quotas and incidents, which makes it a strong failover target for the same model — no change in behaviour to evaluate, unlike falling back to a different model.

It is not free. Model identifiers differ: Bedrock addresses newer Claude models through inference-profile ids with a regional or global. prefix, and a base model id without one can fail with a 400⁷. Feature support differs, as above. Regional endpoints can carry different pricing and data-routing properties⁷. So treat each host as a target with its own model mapping, run your evaluation set against it once, and exercise it in a drill (see failover drills) rather than discovering the differences during an incident.

There is a narrower case worth separating: the same key and the same account reachable through more than one API hostname. Switching hostname when a host is unreachable is not a failover to another account; it should not spend a retry or count against the account's health.

Choosing providers per request

Sometimes the caller knows something the route does not: this request carries data that must not go to an aggregator, or this tenant's contract names one cloud. A per-request allow or deny list of providers handles that without a route per combination. Two rules keep it safe: it can only narrow what the key and the route already allow, and a request whose list leaves nothing able to serve it should fail with an error that says so, not fall back to an unlisted provider.

Record every attempt

A routing policy you cannot observe is one you cannot trust. For each request, record the attempts in order: the target and key, the host when there are several, the status, the provider's own error message, the latency, and why the attempt ended. Then "we failed over" is evidence on a request record, and a policy change can be judged by how many requests needed a second attempt before and after. When the chain is exhausted, the error returned to the caller should name each attempt rather than collapse them into a generic upstream failure.

Routes, rules and failure rules in INFRO

In INFRO a route is a name your application sends as model. It resolves to an ordered list of targets on your own provider connections, each naming a connection and a model. Routes are available on every plan, with no limit on their number, and are edited in the console under Routing. The management API can read them from the Startup plan and write them from the Team plan, with a token holding the routes scope.

A route has a strategy (priority or weighted; a weighted target at weight 0 is a standby that never takes a first attempt), up to 10 targets, a per-attempt timeout_ms (1,000 to 600,000), max_retries on the same target (0 to 3, default 0, run without backoff), up to 20 rules and a failover object.

{
  "slug": "support-agent",
  "name": "Support agent",
  "strategy": "priority",
  "timeout_ms": 60000,
  "max_retries": 1,
  "targets": [
    { "connection_id": "conn_anthropic", "model": "anthropic/claude-sonnet-5.5" },
    { "connection_id": "conn_vertex", "model": "anthropic/claude-sonnet-5.5" },
    { "connection_id": "conn_openai", "model": "openai/gpt-5.6-sol" }
  ],
  "rules": [
    {
      "name": "Enterprise tenants, long prompts",
      "match": "all",
      "conditions": [
        { "field": "metadata", "key": "tier", "op": "in", "value": ["enterprise"] },
        { "field": "input_tokens", "op": "gt", "value": 50000 }
      ],
      "targets": [
        { "connection_id": "conn_vertex", "model": "anthropic/claude-opus-5.5" },
        { "connection_id": "conn_anthropic", "model": "anthropic/claude-opus-5.5" }
      ],
      "fallthrough": true
    }
  ],
  "failover": {
    "rate_limit": "failover",
    "server_error": "retry",
    "timeout": "failover",
    "connection": "failover",
    "credential": "failover",
    "moderation": "stop"
  }
}

Rules. Conditions read metadata (with a key), the attribution fields feature, workflow, step, end_user_id and session_id, the key's environment, project_id and api_key_id, stream, has_images and has_tools, input_tokens (an upper-bound estimate) and max_tokens, and hour_utc and weekday_utc. Operators are eq, neq, in, not_in, contains, starts_with, ends_with, gt, gte, lt, lte, exists and not_exists; a rule holds up to 10 conditions under match: "all" or "any". A value the request did not send is not set: exists is false, neq and not_in are true, every other operator is false. The first matching rule decides. Without fallthrough, a matched rule whose targets cannot serve answers 503. Rules read metadata, never the prompt; the one exception is a route with a prompt classifier you configure, whose rules see only the class it picked. The console's dry run shows which rule a sample request matches.

Failure rules. failover maps each failure kind — rate_limit (429), server_error (5xx), timeout, connection (DNS, TLS or connection failure), credential (401/403) and moderation (a provider's safety refusal) — to retry (the route's retries on the same target, then the next target), failover (the next target at once) or stop (return this error now). Kinds you leave out keep retry. A provider's 400 or 422 on a chat completion is not one of the kinds: the gateway lets the chain continue, because the same request can be valid on one host and not on another, and when every target rejects it the caller gets 400 with the provider's own message. With the default max_retries of 0, retry simply moves on. Route retries run immediately, which is why the example sends rate_limit straight to the next target: retrying a throttled key at once does not help, while one quick retry on a transient 5xx may.

Keys and hosts. A request that names a model directly, rather than a route, is served by every enabled connection whose provider serves that model, providers in the order you connected them, and spread across each provider's active keys. A key marked standby is tried only after every active key of that provider failed; a key that answered 429 goes to the back of the line for the provider's Retry-After, at most five minutes, or 30 seconds when none was sent. Routes name their keys explicitly and ignore the standby setting. Separately, a connection can list up to five alternate API hosts for the same key, tried in order when its host cannot be reached; switching host spends no route retry and does not count toward ejection. Connections that keep failing are ejected for a cool-off that starts at 30 seconds and grows to at most 15 minutes; a 429 never counts toward it.

Per-request providers. routing.providers narrows which of your providers may serve one request, by provider id, on a route's targets, a model's connections and any fallbacks alike. It never widens what the key allows. An unknown id is 400 unknown_provider; a list that leaves nothing able to serve is 503 connection_out_of_scope.

{
  "model": "anthropic/claude-sonnet-5.5",
  "messages": [{ "role": "user", "content": "Summarise this contract clause." }],
  "routing": { "providers": { "deny": ["openrouter"] } }
}

The attempt trail. The chat response names the provider, connection and route that answered. GET /v1/requests/{id} returns fallback_attempts: for each failed attempt, the model, the route position, the error, its latency, the provider and connection (with its name), the host when a connection has several, the upstream status and the reason in the provider's own words. When every target fails, the error names each connection tried. The Routes docs page and Connect a provider have the full contract.