# LLM routing strategies for reliable production AI

URL: https://infro.io/blog/llm-routing-strategies-production
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-08-29
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Routing and reliability
Format: Guide

> Production LLM routing is a policy, not an algorithm: first filter candidates to the ones that are allowed and able to serve the request, then order the rest by a strategy you can explain — usually a fixed priority or a weight — and move to the next only when an attempt fails before the response starts. Changing the provider account is an infrastructure decision; changing the model changes behaviour and needs evaluation.

Key takeaways:
- Separate the two decisions: which provider account serves a model is infrastructure; which model serves a request is product behaviour.
- Filter for eligibility before ranking by preference, so a request can never be sent somewhere it is not allowed or able to go.
- Priority and weighted strategies are easy to reason about during an incident; dynamic scoring needs fresh, trustworthy signals and still flaps.
- Provider-side prompt caches and rate limits belong to an account, so moving traffic between accounts has cost and capacity effects.
- A routing policy is ready when replayed traffic and injected failures produce the outcome you predicted, and the request record explains why.

Routing is the part of an AI system that decides where a request runs. It is tempting to treat it as an optimization problem — score every model on price, speed and quality, pick the best — and that framing produces routing behaviour nobody can predict or explain afterwards. In production, routing is a policy that has to be predictable at 3 a.m., explainable in a postmortem and testable before it ships.

This guide covers the decisions a routing policy makes, five strategies and how each breaks, the signals they depend on, and a method for validating a policy before it carries real traffic.

## Routing is two decisions, not one

Every routed request answers two separate questions, and conflating them is an easy design mistake to make.

|  | Which provider account serves the model | Which model serves the request |
| --- | --- | --- |
| Example | Claude through Anthropic's API or through a cloud platform | A small model for classification, a larger one for drafting |
| What changes for the user | Latency and availability; the model's behaviour should not | Output quality, tone, tool-call reliability, refusals, cost |
| Who should approve a change | The platform or infrastructure owner | The product owner, with evaluation evidence |
| How to test it | Failure injection and latency measurement | An evaluation set with task-specific pass criteria |
| Safe to automate | Usually | Only within limits you have evaluated |

Table: Moving traffic between accounts is an infrastructure change. Moving it between models is a product change.

A policy that keeps these apart can fail over between accounts aggressively while refusing to swap models without evidence — a distinction a single "best model" score cannot express.

## Five strategies, and how each one breaks

### 1. Priority: a primary and ordered standbys

Candidates are tried in a fixed order; traffic moves to the second only when the first fails. It mirrors how general-purpose proxies treat priority levels, where hosts at the highest priority take the traffic and lower levels receive it as higher ones become unhealthy[1].

Use it when you have a clear primary — the account with the best terms or the region closest to your users — and one or more standbys. It is the easiest strategy to reason about during an incident, because "where is traffic going?" has one answer.

**How it breaks:** the standby is never exercised until the day it is needed, so a misconfigured credential or an unmapped model id is discovered during the outage. Send a small share of scheduled test traffic through every standby.

### 2. Weighted: spread traffic on purpose

A first attempt is chosen at random in proportion to weights. Two reasons justify it. Provider rate limits are defined per organization and project[2] and measured as requests and tokens per minute[3], so splitting a high-volume workload across two accounts spreads it across two allowances. And a small weight is a controlled way to canary a change: a partial, time-limited exposure evaluated against the unchanged remainder[4].

**How it breaks:** weights that look balanced are not balanced in cost. Two accounts can carry different negotiated rates, and a model on a cloud platform can bill off a different rate card from the same model on the vendor's own API. Attribute cost per account before you set weights by capacity alone.

### 3. Eligibility filtering: rules before preferences

Before any strategy ranks candidates, remove the ones that must not or cannot serve the request: models without the required context length, tool calling or modality; accounts outside the caller's environment; models an allowlist forbids. A model without image input is ineligible for an image request. A model that is slightly slower is merely less preferred.

**How it breaks:** eligibility encoded as a preference ("prefer models with vision") eventually routes an image to a text model when the preferred ones are down. Hard constraints must be filters, and an empty candidate list must be an explicit error rather than a silent fallback to anything that answers. The same holds for a caller's own constraint — "this request must not go to an aggregator" — sent as a per-request provider list: it may only narrow what the key and route already allow, never widen it.

### 4. Health-aware: stop sending traffic to what is failing

Candidates that are failing are taken out of rotation for a period and brought back when they recover. This is the same idea as outlier detection in service meshes, where a host returning consecutive server errors is ejected for a base time that grows with repeated ejections[5].

**How it breaks:** two ways. Thresholds that are too sensitive make routes flap in and out on ordinary noise; thresholds that are too lax keep sending traffic into an outage. And health signals can be self-inflicted — an account that is healthy but over its rate limit looks "failing" to a naive detector, so the detector should distinguish throttling from errors.

### 5. Evaluation-backed escalation

Start with an efficient model and escalate to a stronger one only on an observable signal: a response that fails schema validation, a failed tool call, a verifier's low score, or a request class you have marked as high-risk. This saves cost without asking a smaller model to imitate a larger one on everything. It is covered in depth in [quality-aware routing](https://infro.io/blog/quality-aware-llm-routing).

**How it breaks:** escalation on the model's own stated confidence, which is not a reliable signal; and escalation logic that lives in only one service, so the next team re-implements it differently.

### Dynamic scoring, hedging and why they come later

Scoring every candidate continuously on recent latency and cost sounds strictly better than a fixed order. In practice it needs signals that are fresh, comparable across providers and resistant to one slow minute, and when it misbehaves nobody can say where the next request will go. Hedged requests — sending a second copy after a short delay and using whichever answers first — were shown to cut tail latency in large services while adding only a small amount of extra load when the delay is chosen carefully[6]. For generative requests, the second copy is billed by the provider, so hedge only traffic whose latency is worth paying twice for.

## The signals a router depends on

| Signal | Where it comes from | Freshness it needs | Failure mode |
| --- | --- | --- | --- |
| Eligibility (capabilities, environment, policy) | Configuration and model metadata | Changes with deploys | Stale metadata sends a request somewhere it cannot run |
| Recent errors and timeouts | The router's own attempts | Seconds to minutes | Too sensitive flaps; too slow keeps traffic in an outage |
| Rate-limit responses | Provider 429s | Seconds | Treated as errors, they eject a healthy but busy account |
| Latency | Attempts, per region | Minutes | Averages hide a tail that users feel |
| Estimated cost | Token counts and the rate for that account | Per request | List prices misrepresent accounts with negotiated rates |
| Quality | Your evaluation results | Per model or prompt release | Absent entirely, so cost wins by default |

Table: Signals ordered from cheapest to hardest to keep trustworthy.

## Interactions that break naive routing

**Prompt caches belong to an account.** Providers discount repeated prompt prefixes — OpenAI applies a reduced cached-input rate automatically on supported models above a minimum prompt length[7], and Anthropic prices cache reads at a fraction of the base input rate while charging more to write the cache[8]. A cache warmed on one account does nothing for a request sent to another. Spreading a cache-heavy workload evenly across accounts can raise both cost and time to first token.

**Retries multiply load at the worst moment.** Every retry is extra traffic sent to a system that is already struggling, and retries at several layers compound[9]. Keep same-target retries few, and let moving to a different candidate be the main recovery path.

**Streaming sets a commit point.** Once part of a response has reached the client, switching candidates would produce duplicated or contradictory output. Routing decisions are only free before the first response event; see [automatic LLM failover](https://infro.io/blog/automatic-llm-failover-guide).

**Model substitution is not failover.** Falling back from one model to a different one changes output length, formatting, tool-call behaviour and refusals. Only allow it where an evaluation says the substitute passes for that workload.

## A worked routing policy

> **Example: A route for a customer-support assistant**
> The assistant drafts replies and calls two tools. The team has an Anthropic account with negotiated rates and a cloud-platform account that can serve the same model family as a standby.
> 
> | Target | Account | Model | Strategy role | Why |
> | --- | --- | --- | --- | --- |
> | 1 | Anthropic, production | The evaluated drafting model | Primary | Best terms, closest region |
> | 2 | Cloud platform, production | The same model family | Standby | Separate account, separate limits and incidents |
> | 3 | OpenAI, production | A different model that passed the support evaluation | Last resort | Different behaviour; allowed only because it passed |
> 
> Per-attempt timeout is set below the client's own deadline so a slow primary leaves time for a standby. No same-target retries, because moving to the next account is the recovery. The route is limited to the production environment, so staging keys cannot reach production accounts.

The table is the artifact to review, not the code. It says where traffic goes, why, and what the team has agreed to accept when the primary fails.

## Validating a policy before it ships

1. **Replay representative traffic** through each candidate and record success rate, cost per successful task, time to first token and malformed-output rate. A cheaper candidate that needs retries is not cheaper.
2. **Inject failures** on purpose: a timeout, a 429, a 5xx and a malformed stream. Check that each produces the outcome you predicted, and that the request record shows every attempt.
3. **Check the empty case.** Disable every candidate and confirm the application receives a clear error rather than an unexpected model.
4. **Canary the change** with a small weight or a small share of traffic, and compare against the unchanged route before widening it.
5. **Write down the rollback.** Reverting a route should be a configuration change that takes effect on the next request.

- [ ] Every target is eligible for the request types the route will receive.
- [ ] Every standby has served real test traffic in the last week.
- [ ] Cross-model targets are backed by a passing evaluation for this workload.
- [ ] The per-attempt timeout leaves room for at least one standby within the client's deadline.
- [ ] Cost per successful task is known for each target at that account's actual rate.
- [ ] The request record for an injected failure names each failed attempt and the target that finally served.

## How routing works in INFRO

INFRO applies these ideas to the provider accounts you connect. A request's `model` field resolves in a fixed order:

1. **A named route.** If `model` matches the slug of a route that serves the key's environment, the route's enabled targets are used. A route has one to ten targets, each a pair of one of your connections and a model, and one of two strategies: `priority`, which tries targets in the order you set, or `weighted`, which chooses the first attempt by weight and tries the rest in descending weight; a target at weight 0 is a standby that never takes a first attempt.
2. **A model id.** Otherwise a model id such as `anthropic/claude-sonnet-5` resolves to every enabled connection in your organization whose provider serves that model: providers in the order you connected them, and within one provider the requests spread across its active keys, with keys you mark standby tried only after every active key failed.
3. **Nothing.** If no connection can serve it, the request fails with a `503` whose message names the provider to connect.

Eligibility comes first, as the strategy sections above argue. Candidates that are disabled, outside the key's environment, or temporarily ejected are skipped before any attempt, and a request can narrow the set further with `routing.providers` (an allow or deny list of provider ids) but never widen it. Ejection follows repeated failures, starts at 30 seconds and grows to at most 15 minutes, and one success clears it; a provider's `429` never counts toward it — the throttled key instead goes to the back of the line for the provider's `Retry-After`. A connection can also list alternate API hosts for the same key, tried when its host cannot be reached, without spending a retry.

Routes go further than a fixed order when you need them to. Up to 20 if-rules pick a different target list from request metadata, the key, project or environment, prompt size, streaming, images, tools or time of day; failure rules decide per failure kind — rate limit, server error, timeout, connection, credential, moderation — whether to retry the same target, move to the next one or stop. For the escalation strategy, a route can have a decision model you connect classify each prompt and let rules send each class to its own targets, with a fallback class when the classifier is unsure or unreachable. [Routing rules and failure rules](https://infro.io/blog/llm-routing-rules-failure-rules) walks through the schema. Each route also sets a per-attempt timeout and zero to three retries on the same target, run without backoff. An attempt can move to the next target only before the response has started. The calling code does not change when the route does:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1", api_key=os.environ["INFRO_API_KEY"])

reply = client.chat.completions.create(
    model="support-drafts",  # the route slug; its targets are configuration, not code
    messages=[{"role": "user", "content": "Draft a reply to this ticket: ..."}],
)
```

The response names the route, the provider kind and the connection that served it, and the stored request record lists every failed attempt with its connection, error type and latency. What INFRO does not do today is score your connections dynamically on latency or cost, or send hedged requests: the order is the one you configured, or the one your rules chose. See [routes](https://infro.io/docs/routes) and [fallbacks](https://infro.io/docs/features/fallbacks) for the full contract.

## Sources

1. Priority levels (Envoy documentation source) — Envoy Project. https://github.com/envoyproxy/envoy/blob/main/docs/root/intro/arch_overview/upstream/load_balancing/priority.rst (accessed 2026-10-13)
2. Rate limits — OpenAI. https://developers.openai.com/api/docs/guides/rate-limits (accessed 2026-10-13)
3. Rate limits — Anthropic. https://platform.claude.com/docs/en/api/rate-limits (accessed 2026-10-13)
4. Canarying Releases — The Site Reliability Workbook — Google. https://sre.google/workbook/canarying-releases/ (accessed 2026-10-13)
5. Outlier detection (Envoy documentation source) — Envoy Project. https://github.com/envoyproxy/envoy/blob/main/docs/root/intro/arch_overview/upstream/outlier.rst (accessed 2026-10-13)
6. The Tail at Scale (Dean and Barroso, Communications of the ACM, 2013) — Google Research. https://research.google/pubs/the-tail-at-scale/ (accessed 2026-10-13)
7. Prompt caching — OpenAI. https://developers.openai.com/api/docs/guides/prompt-caching (accessed 2026-10-13)
8. Prompt caching — Anthropic. https://platform.claude.com/docs/en/build-with-claude/prompt-caching (accessed 2026-10-13)
9. Timeouts, retries, and backoff with jitter — Amazon Builders' Library. https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter (accessed 2026-10-13)

## Revision history

- 2026-10-13: Reviewed against every source and the current gateway. Eligibility filtering now covers per-request provider lists. INFRO's section is updated: requests that name a model are spread across each provider's keys with standby keys last, routes take if-rules, failure rules and a prompt classifier, connections can list alternate hosts, and the ejection window and 429 handling are stated. The hedged-requests citation now points to the paper's Google Research page.
- 2026-09-11: Expanded from a short overview into a full guide: the model-versus-provider distinction, five strategies with their failure modes, signals, caching and rate-limit interactions, a validation method, a worked route, and how INFRO routes are actually resolved. Added cited sources.