# LLM routing rules: if-rules, failure rules, key spreading

URL: https://infro.io/blog/llm-routing-rules-failure-rules
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-13
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Routing and reliability
Format: Guide

> Declarative LLM routing separates eligibility (who may serve a request), selection (if-rules that match on metadata, key, size or time and pick a target list) and execution (failure rules that decide, per failure kind, whether to retry the same target, move to the next one, or stop). Fail over on rate limits, server errors and unreachable hosts; stop on requests the provider rejected as invalid unless hosts genuinely differ in what they accept; treat timeouts and safety refusals as explicit policy choices. Spread load across several keys of one provider with a standby key held back, and record every attempt so recovery is evidence rather than an assumption.

Key takeaways:
- Split routing into eligibility, selection and execution: filters only ever remove candidates, rules pick a target list, failure rules decide what happens after each failed attempt.
- Match rules on request metadata — tenant tier, feature, key, project, size, streaming, tools, time — never on prompt text, and decide explicitly what a request that sets nothing should match.
- Classify every failure before acting: a 429 means try another key or account, a 5xx means try another target, a rejected request usually means stop, and timeouts and safety refusals need a policy per workload.
- Several keys of one provider are separate rate-limit windows: spread requests across them, cool a key that returned 429 for its Retry-After, and hold one key as a standby for failures only.
- The same model on different hosts is a separate failure domain with different model ids, features and quotas; treat it as a target, and test it, before you need it.

On 3 September 2026, Anthropic's status page recorded elevated errors on several Claude models from 13:26 UTC, narrowing over the afternoon to Opus 4.8 and Opus 5, with impact ending at 16:16 UTC; a shorter incident on Claude Sonnet 5 had run from 12:37 to 12:56[1]. The same day, OpenAI's status page recorded elevated errors across ChatGPT and Codex[2]. Incidents like these are rarely "the provider is down". They are usually some models, on some endpoints, for some hours — which is exactly the shape a routing policy can respond to, if the policy exists before the incident does.

This guide is about writing that policy down: the conditions that pick targets, what each kind of failure should do, how to spread traffic across several keys and hosts, and what to record. It builds on [LLM routing strategies](https://infro.io/blog/llm-routing-strategies-production), which covers priority, weighted and health-aware ordering, and on [Automatic LLM failover](https://infro.io/blog/automatic-llm-failover-guide), which covers commit points and idempotency. The last section shows INFRO's route schema with JSON you can send.

## Routing as data, not as a retry loop

Most routing starts as a loop in application code: try provider A, catch, try provider B. It works until someone needs to change it during an incident, or asks why a request took 40 seconds, or discovers that the loop retried a malformed request across four providers. A declarative policy moves those decisions into configuration that can be read, reviewed, versioned and changed without a deploy.

A useful policy answers three questions, in order:

[Diagram: The three decisions in a routing policy, plus the record it leaves. Filters only remove candidates; rules choose a list; failure rules act on each attempt.]

Keeping them separate prevents a common class of bug. An allowlist that a routing rule can widen is not an allowlist. A per-request hint that can add a provider the key was never allowed to reach is a privilege escalation. Eligibility filters should only ever remove candidates, and everything after them should operate on what is left.

## Match conditions: what a rule may read

An if-rule is a set of conditions and a target list: when the conditions hold, those targets serve the request instead of the default list. The useful conditions are facts about the request that the caller declares or the gateway knows:

- **Who and what for.** Tenant tier from request metadata, the product feature, workflow or step, the end user, the API key, the project, the environment.
- **Shape.** Estimated prompt size, `max_tokens`, whether it streams, whether it carries images or declares tools.
- **When.** Hour of day and weekday, for batch windows or regional business hours.

What a rule should not read is the prompt text. Rules that pattern-match on content turn routing configuration into a place where customer content is inspected and logged, and they break silently when someone rewords a system prompt. If you need content-aware routing — small prompts to a small model, hard ones to a large one — make it an explicit classification step whose output is a label, and let rules match the label. The [quality-aware routing guide](https://infro.io/blog/quality-aware-llm-routing) covers that design.

Three semantics are worth deciding deliberately, because they are where rule sets go wrong:

1. **First match wins.** Evaluate rules top to bottom and stop at the first that holds. Overlapping rules are then resolved by order, which a reviewer can see, rather than by specificity scores nobody can predict.
2. **"Not set" is its own value.** A request that sends no `tier` should not match `tier != "free"`. A good evaluator treats a missing value as matching only `not_exists`, `neq` and `not_in`, so a request that says nothing never lands on a rule written for requests that say something specific — or, if your policy wants the opposite, says so explicitly.
3. **Fall through, or not.** When a matched rule's targets all fail, either continue into the default list or stop. Stopping is right when the rule exists to keep traffic *off* the default targets — an EU tenant that must stay on EU-hosted connections, say. Falling through is right when the rule merely prefers a target.

Keep rule sets short. Twenty rules nobody can hold in their head are a configuration incident waiting to happen; a dry run that shows which rule a sample request matches, and why, is worth more than another rule.

## Classify the failure before you act on it

The single most important property of a failover policy is that it treats different failures differently. Providers make the distinctions explicit. Anthropic documents 429 `rate_limit_error` for an organization's rate limits or spend caps, 529 `overloaded_error` for API-wide load, 500 `api_error` for internal failures, 401 and 403 for credential and permission problems, and 400 `invalid_request_error` for problems with the request itself[3]. OpenAI splits its 429s into rate limits, where the advice is to honour `Retry-After`, and quota or billing limits, where "retrying won't restore access"[4]. RFC 6585 lets a 429 carry a `Retry-After` header but does not require it[5].

[Diagram: What each failure kind says about the target, and whether trying it again or trying elsewhere can help. Error semantics from Anthropic's and OpenAI's error documentation, read 2026-10-13.]

Some of these deserve more than a cell:

- **429 is about one key or account, not the provider.** Trying the same key again before its window resets only adds to the queue. A different key, project or account has its own window and can carry the request now. A rate limit should never count as a health failure, or a busy-but-healthy account gets ejected. The [circuit breaker guide](https://infro.io/blog/llm-circuit-breakers-throttling) goes into that distinction.
- **Quota and spend-cap 429s are a stop for that account.** Anthropic notes that a tier spend-cap 429 has no `retry-after` header "and keeps failing until access resumes"[3]. Move to another account if you have one; otherwise fail fast with a clear message.
- **Timeouts are ambiguous.** The provider may still be generating, and billing, the answer you gave up on. Retrying a long generation can mean paying for it twice. For interactive traffic, failing over is usually worth it; for long batch generations, a single attempt with a generous timeout can be the better policy.
- **Safety refusals are a governance decision.** Each provider applies its own policy, so another provider may accept what one refused. Whether that is acceptable is not a reliability question; decide it per workload, and make the default conservative.

## Why not to fail over on client errors

A request the provider rejected as malformed — an unknown parameter, an invalid schema, a context window exceeded — will usually be rejected by the next target too. Failing over then costs latency on every hop, and it multiplies load in exactly the moment a bug has made every request bad. The AWS Builders' Library puts the amplification starkly: five layers each retrying three times can put 243 times the load on the component at the bottom[6]. Its guidance is to retry at one layer only, and not to retry errors caused by the request itself[6].

There is one real exception, and it is specific to multi-host LLM routing: the same model on different hosts does not accept the same requests. Anthropic's own guide for Claude on Amazon Bedrock lists features Bedrock does not support, including URL sources for images and documents, server-side tools, and the API's server-side `fallbacks` parameter[7]. A request using one of those is a valid request on the Anthropic API and an invalid one on Bedrock. If your targets span hosts, a 400 can be host-specific, and the policy has to choose between "stop" (fast, honest) and "try the next host" (more likely to succeed, slower when the request is genuinely bad). Either way, return the provider's own error message to the caller, so a bad request is visibly bad.

## Spreading load across several keys

Rate limits are scoped to keys, projects, workspaces or accounts, depending on the provider. Several keys of one provider are therefore several windows, and a request that one key cannot serve right now another can. The pattern:

- **Spread, do not pin.** Share requests across the active keys of a provider rather than sending everything to the first one until it fails.
- **Cool a throttled key.** After a 429, move that key to the back of the line for its `Retry-After`, with a bound and a default when the provider sent none.
- **Hold a standby.** Mark one key as standby so it takes traffic only after every active key has failed. Its window is then fresh when you need it, which is the point.
- **Name the keys.** A key named "OpenAI batch" or "Anthropic standby" makes the trace readable; "key 3" does not.

[Diagram: Requests spread across two active keys; a 429 sends the same request to the other key and cools the first; the standby key is reached only when both active keys fail. Illustrative sequence.]

## Same model, different hosts

Many models are served by more than one host: Claude through Anthropic's API, Amazon Bedrock and Google Cloud's Vertex AI; OpenAI models through OpenAI and Azure. Each host is a separate failure domain with its own quotas and incidents, which makes it a strong failover target for the *same* model — no change in behaviour to evaluate, unlike falling back to a different model.

It is not free. Model identifiers differ: Bedrock addresses newer Claude models through inference-profile ids with a regional or `global.` prefix, and a base model id without one can fail with a 400[7]. Feature support differs, as above. Regional endpoints can carry different pricing and data-routing properties[7]. So treat each host as a target with its own model mapping, run your evaluation set against it once, and exercise it in a drill (see [failover drills](https://infro.io/blog/llm-failover-drills)) rather than discovering the differences during an incident.

There is a narrower case worth separating: the same key and the same account reachable through more than one API hostname. Switching hostname when a host is unreachable is not a failover to another account; it should not spend a retry or count against the account's health.

## Choosing providers per request

Sometimes the caller knows something the route does not: this request carries data that must not go to an aggregator, or this tenant's contract names one cloud. A per-request allow or deny list of providers handles that without a route per combination. Two rules keep it safe: it can only narrow what the key and the route already allow, and a request whose list leaves nothing able to serve it should fail with an error that says so, not fall back to an unlisted provider.

## Record every attempt

A routing policy you cannot observe is one you cannot trust. For each request, record the attempts in order: the target and key, the host when there are several, the status, the provider's own error message, the latency, and why the attempt ended. Then "we failed over" is evidence on a request record, and a policy change can be judged by how many requests needed a second attempt before and after. When the chain is exhausted, the error returned to the caller should name each attempt rather than collapse them into a generic upstream failure.

## Routes, rules and failure rules in INFRO

In INFRO a route is a name your application sends as `model`. It resolves to an ordered list of targets on your own provider connections, each naming a connection and a model. Routes are available on every plan, with no limit on their number, and are edited in the console under Routing. The [management API](https://infro.io/docs/manage/resources) can read them from the Startup plan and write them from the Team plan, with a token holding the `routes` scope.

A route has a `strategy` (`priority` or `weighted`; a weighted target at weight 0 is a standby that never takes a first attempt), up to 10 `targets`, a per-attempt `timeout_ms` (1,000 to 600,000), `max_retries` on the same target (0 to 3, default 0, run without backoff), up to 20 `rules` and a `failover` object.

```json
{
  "slug": "support-agent",
  "name": "Support agent",
  "strategy": "priority",
  "timeout_ms": 60000,
  "max_retries": 1,
  "targets": [
    { "connection_id": "conn_anthropic", "model": "anthropic/claude-sonnet-5.5" },
    { "connection_id": "conn_vertex", "model": "anthropic/claude-sonnet-5.5" },
    { "connection_id": "conn_openai", "model": "openai/gpt-5.6-sol" }
  ],
  "rules": [
    {
      "name": "Enterprise tenants, long prompts",
      "match": "all",
      "conditions": [
        { "field": "metadata", "key": "tier", "op": "in", "value": ["enterprise"] },
        { "field": "input_tokens", "op": "gt", "value": 50000 }
      ],
      "targets": [
        { "connection_id": "conn_vertex", "model": "anthropic/claude-opus-5.5" },
        { "connection_id": "conn_anthropic", "model": "anthropic/claude-opus-5.5" }
      ],
      "fallthrough": true
    }
  ],
  "failover": {
    "rate_limit": "failover",
    "server_error": "retry",
    "timeout": "failover",
    "connection": "failover",
    "credential": "failover",
    "moderation": "stop"
  }
}
```

**Rules.** Conditions read `metadata` (with a `key`), the attribution fields `feature`, `workflow`, `step`, `end_user_id` and `session_id`, the key's `environment`, `project_id` and `api_key_id`, `stream`, `has_images` and `has_tools`, `input_tokens` (an upper-bound estimate) and `max_tokens`, and `hour_utc` and `weekday_utc`. Operators are `eq`, `neq`, `in`, `not_in`, `contains`, `starts_with`, `ends_with`, `gt`, `gte`, `lt`, `lte`, `exists` and `not_exists`; a rule holds up to 10 conditions under `match: "all"` or `"any"`. A value the request did not send is not set: `exists` is false, `neq` and `not_in` are true, every other operator is false. The first matching rule decides. Without `fallthrough`, a matched rule whose targets cannot serve answers `503`. Rules read metadata, never the prompt; the one exception is a route with a prompt classifier you configure, whose rules see only the class it picked. The console's dry run shows which rule a sample request matches.

**Failure rules.** `failover` maps each failure kind — `rate_limit` (429), `server_error` (5xx), `timeout`, `connection` (DNS, TLS or connection failure), `credential` (401/403) and `moderation` (a provider's safety refusal) — to `retry` (the route's retries on the same target, then the next target), `failover` (the next target at once) or `stop` (return this error now). Kinds you leave out keep `retry`. A provider's 400 or 422 on a chat completion is not one of the kinds: the gateway lets the chain continue, because the same request can be valid on one host and not on another, and when every target rejects it the caller gets `400` with the provider's own message. With the default `max_retries` of 0, `retry` simply moves on. Route retries run immediately, which is why the example sends `rate_limit` straight to the next target: retrying a throttled key at once does not help, while one quick retry on a transient 5xx may.

**Keys and hosts.** A request that names a model directly, rather than a route, is served by every enabled connection whose provider serves that model, providers in the order you connected them, and spread across each provider's active keys. A key marked **standby** is tried only after every active key of that provider failed; a key that answered `429` goes to the back of the line for the provider's `Retry-After`, at most five minutes, or 30 seconds when none was sent. Routes name their keys explicitly and ignore the standby setting. Separately, a connection can list up to five alternate API hosts for the same key, tried in order when its host cannot be reached; switching host spends no route retry and does not count toward ejection. Connections that keep failing are ejected for a cool-off that starts at 30 seconds and grows to at most 15 minutes; a 429 never counts toward it.

**Per-request providers.** `routing.providers` narrows which of your providers may serve one request, by provider id, on a route's targets, a model's connections and any `fallbacks` alike. It never widens what the key allows. An unknown id is `400 unknown_provider`; a list that leaves nothing able to serve is `503 connection_out_of_scope`.

```json
{
  "model": "anthropic/claude-sonnet-5.5",
  "messages": [{ "role": "user", "content": "Summarise this contract clause." }],
  "routing": { "providers": { "deny": ["openrouter"] } }
}
```

**The attempt trail.** The chat response names the `provider`, `connection` and `route` that answered. `GET /v1/requests/{id}` returns `fallback_attempts`: for each failed attempt, the model, the route position, the error, its latency, the provider and connection (with its name), the host when a connection has several, the upstream status and the reason in the provider's own words. When every target fails, the error names each connection tried. The [Routes docs page](https://infro.io/docs/routes) and [Connect a provider](https://infro.io/docs/providers) have the full contract.

## Sources

1. Claude status: incident history feed — Anthropic. https://status.claude.com/history.rss (accessed 2026-10-13)
2. OpenAI status: incident history feed — OpenAI. https://status.openai.com/history.rss (accessed 2026-10-13)
3. Claude API errors — Anthropic. https://platform.claude.com/docs/en/api/errors (accessed 2026-10-13)
4. Error codes — OpenAI. https://developers.openai.com/api/docs/guides/error-codes (accessed 2026-10-13)
5. RFC 6585: Additional HTTP Status Codes — IETF. https://www.rfc-editor.org/rfc/rfc6585.html (accessed 2026-10-13)
6. Timeouts, retries and backoff with jitter — Amazon Web Services. https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter (accessed 2026-10-13)
7. Claude on Amazon Bedrock — Anthropic. https://platform.claude.com/docs/en/build-with-claude/claude-on-amazon-bedrock-legacy (accessed 2026-10-13)