Routing is the part of an AI system that decides where a request runs. It is tempting to treat it as an optimization problem — score every model on price, speed and quality, pick the best — and that framing produces routing behaviour nobody can predict or explain afterwards. In production, routing is a policy that has to be predictable at 3 a.m., explainable in a postmortem and testable before it ships.
This guide covers the decisions a routing policy makes, five strategies and how each breaks, the signals they depend on, and a method for validating a policy before it carries real traffic.
Routing is two decisions, not one
Every routed request answers two separate questions, and conflating them is an easy design mistake to make.
| Which provider account serves the model | Which model serves the request | |
|---|---|---|
| Example | Claude through Anthropic's API or through a cloud platform | A small model for classification, a larger one for drafting |
| What changes for the user | Latency and availability; the model's behaviour should not | Output quality, tone, tool-call reliability, refusals, cost |
| Who should approve a change | The platform or infrastructure owner | The product owner, with evaluation evidence |
| How to test it | Failure injection and latency measurement | An evaluation set with task-specific pass criteria |
| Safe to automate | Usually | Only within limits you have evaluated |
A policy that keeps these apart can fail over between accounts aggressively while refusing to swap models without evidence — a distinction a single "best model" score cannot express.
Five strategies, and how each one breaks
1. Priority: a primary and ordered standbys
Candidates are tried in a fixed order; traffic moves to the second only when the first fails. It mirrors how general-purpose proxies treat priority levels, where hosts at the highest priority take the traffic and lower levels receive it as higher ones become unhealthy1.
Use it when you have a clear primary — the account with the best terms or the region closest to your users — and one or more standbys. It is the easiest strategy to reason about during an incident, because "where is traffic going?" has one answer.
How it breaks: the standby is never exercised until the day it is needed, so a misconfigured credential or an unmapped model id is discovered during the outage. Send a small share of scheduled test traffic through every standby.
2. Weighted: spread traffic on purpose
A first attempt is chosen at random in proportion to weights. Two reasons justify it. Provider rate limits are defined per organization and project2 and measured as requests and tokens per minute3, so splitting a high-volume workload across two accounts spreads it across two allowances. And a small weight is a controlled way to canary a change: a partial, time-limited exposure evaluated against the unchanged remainder4.
How it breaks: weights that look balanced are not balanced in cost. Two accounts can carry different negotiated rates, and a model on a cloud platform can bill off a different rate card from the same model on the vendor's own API. Attribute cost per account before you set weights by capacity alone.
3. Eligibility filtering: rules before preferences
Before any strategy ranks candidates, remove the ones that must not or cannot serve the request: models without the required context length, tool calling or modality; accounts outside the caller's environment; models an allowlist forbids. A model without image input is ineligible for an image request. A model that is slightly slower is merely less preferred.
How it breaks: eligibility encoded as a preference ("prefer models with vision") eventually routes an image to a text model when the preferred ones are down. Hard constraints must be filters, and an empty candidate list must be an explicit error rather than a silent fallback to anything that answers. The same holds for a caller's own constraint — "this request must not go to an aggregator" — sent as a per-request provider list: it may only narrow what the key and route already allow, never widen it.
4. Health-aware: stop sending traffic to what is failing
Candidates that are failing are taken out of rotation for a period and brought back when they recover. This is the same idea as outlier detection in service meshes, where a host returning consecutive server errors is ejected for a base time that grows with repeated ejections5.
How it breaks: two ways. Thresholds that are too sensitive make routes flap in and out on ordinary noise; thresholds that are too lax keep sending traffic into an outage. And health signals can be self-inflicted — an account that is healthy but over its rate limit looks "failing" to a naive detector, so the detector should distinguish throttling from errors.
5. Evaluation-backed escalation
Start with an efficient model and escalate to a stronger one only on an observable signal: a response that fails schema validation, a failed tool call, a verifier's low score, or a request class you have marked as high-risk. This saves cost without asking a smaller model to imitate a larger one on everything. It is covered in depth in quality-aware routing.
How it breaks: escalation on the model's own stated confidence, which is not a reliable signal; and escalation logic that lives in only one service, so the next team re-implements it differently.
Dynamic scoring, hedging and why they come later
Scoring every candidate continuously on recent latency and cost sounds strictly better than a fixed order. In practice it needs signals that are fresh, comparable across providers and resistant to one slow minute, and when it misbehaves nobody can say where the next request will go. Hedged requests — sending a second copy after a short delay and using whichever answers first — were shown to cut tail latency in large services while adding only a small amount of extra load when the delay is chosen carefully6. For generative requests, the second copy is billed by the provider, so hedge only traffic whose latency is worth paying twice for.
The signals a router depends on
| Signal | Where it comes from | Freshness it needs | Failure mode |
|---|---|---|---|
| Eligibility (capabilities, environment, policy) | Configuration and model metadata | Changes with deploys | Stale metadata sends a request somewhere it cannot run |
| Recent errors and timeouts | The router's own attempts | Seconds to minutes | Too sensitive flaps; too slow keeps traffic in an outage |
| Rate-limit responses | Provider 429s | Seconds | Treated as errors, they eject a healthy but busy account |
| Latency | Attempts, per region | Minutes | Averages hide a tail that users feel |
| Estimated cost | Token counts and the rate for that account | Per request | List prices misrepresent accounts with negotiated rates |
| Quality | Your evaluation results | Per model or prompt release | Absent entirely, so cost wins by default |
Interactions that break naive routing
Prompt caches belong to an account. Providers discount repeated prompt prefixes — OpenAI applies a reduced cached-input rate automatically on supported models above a minimum prompt length7, and Anthropic prices cache reads at a fraction of the base input rate while charging more to write the cache8. A cache warmed on one account does nothing for a request sent to another. Spreading a cache-heavy workload evenly across accounts can raise both cost and time to first token.
Retries multiply load at the worst moment. Every retry is extra traffic sent to a system that is already struggling, and retries at several layers compound9. Keep same-target retries few, and let moving to a different candidate be the main recovery path.
Streaming sets a commit point. Once part of a response has reached the client, switching candidates would produce duplicated or contradictory output. Routing decisions are only free before the first response event; see automatic LLM failover.
Model substitution is not failover. Falling back from one model to a different one changes output length, formatting, tool-call behaviour and refusals. Only allow it where an evaluation says the substitute passes for that workload.
A worked routing policy
The table is the artifact to review, not the code. It says where traffic goes, why, and what the team has agreed to accept when the primary fails.
Validating a policy before it ships
- Replay representative traffic through each candidate and record success rate, cost per successful task, time to first token and malformed-output rate. A cheaper candidate that needs retries is not cheaper.
- Inject failures on purpose: a timeout, a 429, a 5xx and a malformed stream. Check that each produces the outcome you predicted, and that the request record shows every attempt.
- Check the empty case. Disable every candidate and confirm the application receives a clear error rather than an unexpected model.
- Canary the change with a small weight or a small share of traffic, and compare against the unchanged route before widening it.
- Write down the rollback. Reverting a route should be a configuration change that takes effect on the next request.
- Every target is eligible for the request types the route will receive.
- Every standby has served real test traffic in the last week.
- Cross-model targets are backed by a passing evaluation for this workload.
- The per-attempt timeout leaves room for at least one standby within the client's deadline.
- Cost per successful task is known for each target at that account's actual rate.
- The request record for an injected failure names each failed attempt and the target that finally served.
How routing works in INFRO
INFRO applies these ideas to the provider accounts you connect. A request's model field resolves in a fixed order:
- A named route. If
modelmatches the slug of a route that serves the key's environment, the route's enabled targets are used. A route has one to ten targets, each a pair of one of your connections and a model, and one of two strategies:priority, which tries targets in the order you set, orweighted, which chooses the first attempt by weight and tries the rest in descending weight; a target at weight 0 is a standby that never takes a first attempt. - A model id. Otherwise a model id such as
anthropic/claude-sonnet-5resolves to every enabled connection in your organization whose provider serves that model: providers in the order you connected them, and within one provider the requests spread across its active keys, with keys you mark standby tried only after every active key failed. - Nothing. If no connection can serve it, the request fails with a
503whose message names the provider to connect.
Eligibility comes first, as the strategy sections above argue. Candidates that are disabled, outside the key's environment, or temporarily ejected are skipped before any attempt, and a request can narrow the set further with routing.providers (an allow or deny list of provider ids) but never widen it. Ejection follows repeated failures, starts at 30 seconds and grows to at most 15 minutes, and one success clears it; a provider's 429 never counts toward it — the throttled key instead goes to the back of the line for the provider's Retry-After. A connection can also list alternate API hosts for the same key, tried when its host cannot be reached, without spending a retry.
Routes go further than a fixed order when you need them to. Up to 20 if-rules pick a different target list from request metadata, the key, project or environment, prompt size, streaming, images, tools or time of day; failure rules decide per failure kind — rate limit, server error, timeout, connection, credential, moderation — whether to retry the same target, move to the next one or stop. For the escalation strategy, a route can have a decision model you connect classify each prompt and let rules send each class to its own targets, with a fallback class when the classifier is unsure or unreachable. Routing rules and failure rules walks through the schema. Each route also sets a per-attempt timeout and zero to three retries on the same target, run without backoff. An attempt can move to the next target only before the response has started. The calling code does not change when the route does:
import os
from openai import OpenAI
client = OpenAI(base_url="https://api.infro.io/v1", api_key=os.environ["INFRO_API_KEY"])
reply = client.chat.completions.create(
model="support-drafts", # the route slug; its targets are configuration, not code
messages=[{"role": "user", "content": "Draft a reply to this ticket: ..."}],
)The response names the route, the provider kind and the connection that served it, and the stored request record lists every failed attempt with its connection, error type and latency. What INFRO does not do today is score your connections dynamically on latency or cost, or send hedged requests: the order is the one you configured, or the one your rules chose. See routes and fallbacks for the full contract.