An AI router decides, for each request your application makes, which model should answer it and which provider account should serve that model. It reads facts about the request (the task, its size, the customer tier, whether it streams) and facts about the providers (health, cost, rate limits), and returns an ordered list of targets to try. Done well, it is how you change which model or provider handles a workload without shipping code.
A note on the term: "AI router" is also used for consumer Wi-Fi routers with machine-learning features. This guide is about the other meaning, the component that routes requests between AI models and providers, sometimes called an AI model router or LLM router.
What does an AI router actually decide?
Two separate decisions hide inside "routing", and keeping them apart makes every later choice easier.
- Which model. A short classification prompt and a multi-step analysis do not need the same model. Model routing changes the answer, its cost and its latency.
- Which provider account. Many models are served by more than one provider: a vendor's own API, a cloud platform, an aggregator, or several keys on one account. Provider routing should not change the answer at all; it changes availability, rate-limit headroom and which contract the usage lands on.
Most outage work is provider routing. Most cost and quality work is model routing. A router that blurs the two tends to "fail over" from one model to a different one during an incident, which turns an availability problem into a quality problem nobody chose.
The inputs to a routing decision
A router can only be as good as the facts it can see when the request arrives. These are the ones that matter in production:
| Input | Where it comes from | Typical use |
|---|---|---|
| Task or feature | Request metadata, a header, the key or project | Send each feature to the model it was evaluated on |
| Request size | Estimated prompt tokens, max_tokens | Long prompts to models with room; short ones to small models |
| Cost | Rates per model and per account | Keep low-value traffic on the least expensive acceptable model |
| Latency | Observed time to first token per target | Interactive traffic to fast targets, batch traffic anywhere |
| Quality | Offline evals, online feedback, a classifier's judgment | Hard prompts to strong models |
| Availability | Rate-limit headroom, recent errors | Skip targets that are failing or throttled |
| Data policy | Customer tier, region, model allowlists | Keep regulated traffic on approved models and accounts |
| Provider health | Error rates and timeouts across all traffic | Eject a failing account for a while instead of retrying it on every request |
The last row deserves attention. Health is a property of the account, learned from all traffic, not of the single request. Load balancers have handled this for years: Envoy's outlier detection ejects a host after a configured number of consecutive 5xx responses and lengthens each repeated ejection1. An AI router needs the same memory, or each request rediscovers the same failing provider.
The five kinds of model routing
Static rules
A rule maps something known about the request to targets: "requests from the support feature go to model A", "gold-tier customers go to account B", "prompts over 50,000 tokens go to a long-context model". Rules are predictable, easy to review and easy to explain in an incident. They cover more cases than people expect, because the application usually already knows what kind of work it is asking for. See routing rules and failure rules for the design of conditions that fail safely when a field is missing.
Weighted splits
A weighted route sends a share of traffic to each target: 95% to the current model, 5% to a candidate. It is how you canary a model change or spread load across two accounts. A well-built split still fails over: the weight decides which target is tried first, not which is the only one tried.
Fallback chains
An ordered list: try the primary, and on a timeout, a rate limit or a 5xx, try the next. This is the routing kind that protects availability, covered in depth in the automatic LLM failover guide. The important constraint is timing. Once the first part of a streamed answer has reached the client, moving to another target would produce a duplicated or contradictory response, so a fallback chain only acts before the response starts.
Classifier or semantic routing
A small model reads the prompt and labels it, for example simple or complex, or billing versus technical, and rules send each label to its own targets. Semantic routers do the same with embeddings and nearest-neighbour matching against example prompts. This is the routing kind that looks at content, and it carries costs that the others do not: an extra call per request, extra latency, and a classifier that can be wrong or unavailable. It needs a fallback label for when the classifier times out or is unsure. Classifiers in the request path covers the latency and failure budget.
Cost- and quality-aware cascades
A cascade tries an inexpensive model first and escalates to a stronger one only when the first answer looks inadequate; a learned router predicts in advance which model a prompt needs. The research results are real but specific to their benchmarks. The RouteLLM authors report routers trained on human preference data that cut cost by more than 2x in some cases without degrading quality on the benchmarks they used2. FrugalGPT reports a cascade that matched the best single model's quality at a fraction of its cost on the tasks it studied3. Neither result transfers to your workload without your own evaluation set, which is the subject of quality-aware LLM routing.
| Kind | Reads the prompt | Extra calls | Best for | Main risk |
|---|---|---|---|---|
| Static rules | No | None | Known features, tiers, sizes | Rules nobody revisits |
| Weighted split | No | None | Canaries, migrations, load spreading | Comparing groups without a quality metric |
| Fallback chain | No | Only on failure | Availability | Falling back to a different model by accident |
| Classifier / semantic | Yes | One per request | Mixed traffic on one endpoint | Misclassification, added latency |
| Cascade / learned | Yes, or the answer | Sometimes two | High volume with a clear quality signal | Silent quality loss |
LLM routing strategies for production compares these patterns with worked configurations.
Where should the AI router live?
There are three places to put the decision, and the choice matters more than the routing algorithm.
- A library inside the application. Fast to start and adds no network hop. But each service carries its own copy of the rules, its own provider keys and its own health state. Changing a route means a deploy of every service that uses it, and nothing guarantees they agree.
- A proxy or gateway in the request path. Every caller sends requests to one endpoint, and the router decides there. Rules change in configuration, health is learned from all traffic at once, and every request is recorded with the target that served it. The cost is one more network hop and a service whose availability is now part of yours.
- A control plane. The gateway's router plus the things routing has to respect: budgets that refuse a request before it reaches a provider, model allowlists a route cannot step around, environments, and per-route cost and quality reporting. Routing decisions are only as safe as the policy they cannot violate, and a control plane keeps the two in one place.
A useful test: if a security reviewer asks "can any route send production traffic to a model we have not approved?", where would you look to answer? If the answer is "in every repository", the router is in the wrong place.
What to measure on every route
A router that is not measured will drift. Record these per route and per target, and look at them before and after every change:
- Served model and account. The share of requests each target actually served, not the share it was configured for.
- Failover rate. Attempts that failed before another target answered, by failure kind (rate limit, timeout, server error, credential).
- Latency. Time to first token and total time, at p50 and p95, per target.
- Cost per successful task. Estimated provider cost divided by tasks that succeeded, not by requests. A less expensive model that needs two retries is not less expensive.
- A quality signal. Eval scores on a fixed set, user feedback, or a downstream success metric such as "ticket resolved without escalation".
- Classifier outcomes, where you use one: the share of each label, the share that fell back because the classifier was unsure or slow.
- Prompt-cache hit rate, which routing affects more than most teams expect (next section).
How AI routing fails quietly
Routing failures rarely raise errors. They show up weeks later as a higher bill or a lower satisfaction score.
Routing drift
The rules stay the same while the traffic changes. A new feature starts sending metadata the rules do not recognize, so all of it lands on the default target. A weighted canary set to 5% is forgotten at 50%. A classifier trained on last quarter's prompts labels this quarter's differently. The defence is the measurement above, reviewed on a schedule, and rules that make "matched nothing" visible.
Silent quality regressions
A fallback chain that ends in a weaker model keeps the service up during an outage, but every request served by the last target is a quality change. A cascade that escalates too rarely lowers quality while its cost chart looks excellent. Neither produces an error code. Record which target served each request, and alert when the share served by a fallback target rises.
Prompt-cache misses
Providers cache prompt prefixes, and routing decides whether a request can reuse them. OpenAI requires the entire rendered prefix to match and notes that cache entries live on individual machines, so how requests are routed affects hits4. Anthropic's caches are isolated per organization, and per workspace on its own API, with a default lifetime of five minutes, refreshed on each use5. Splitting one conversation's turns across two providers, two accounts or two models means each target warms its own cache and neither stays warm. Keep a conversation on one target where you can, and keep provider failover between accounts for genuine failures rather than routine load spreading. Then compare cache hit rates before and after a routing change.
Retrying what will not succeed
A 400 for an invalid parameter will fail on the next target too if that target speaks the same dialect; a 401 means a key is wrong, not that the provider is down. Treat failure kinds differently: retry rate limits, move on after server errors, and stop on errors that no other target will fix.
An example route
Here is the shape of a route in INFRO, the BYOK control plane this blog belongs to, as documented in its route reference. The application sends "model": "support-agent"; the route classifies each prompt with a decision model on the customer's own connection, sends hard prompts to a large model, and keeps everything else on the default targets with failover between two of the customer's own accounts.
{
"slug": "support-agent",
"strategy": "priority",
"targets": [
{ "connection_id": "conn_anthropic_primary", "model": "anthropic/claude-haiku-4.5" },
{ "connection_id": "conn_anthropic_standby", "model": "anthropic/claude-haiku-4.5" }
],
"classifier": {
"model": "typesafe/jev-1.13",
"instructions": "How much reasoning does answering this request need?",
"classes": [
{ "key": "small", "description": "A lookup, a greeting or a short factual answer." },
{ "key": "large", "description": "Multi-step reasoning, hard code or careful analysis." }
],
"input": "last_user",
"timeout_ms": 1500,
"min_confidence": 0.5,
"fallback_class": "large"
},
"rules": [
{
"name": "Hard prompts",
"conditions": [{ "field": "class", "op": "eq", "value": "large" }],
"targets": [{ "connection_id": "conn_anthropic_primary", "model": "anthropic/claude-opus-5.5" }],
"fallthrough": true
}
],
"failover": { "rate_limit": "retry", "server_error": "failover", "timeout": "failover" }
}Three details carry the advice above into configuration. Both default targets serve the same model, so provider failover never changes the answer. The classifier has a fallback_class, so a slow or unsure classifier sends the request to the strong model rather than blocking it. And failure rules decide per failure kind whether to retry the same target, move on, or stop.
How INFRO runs the router
INFRO puts the router in the gateway, in front of provider accounts the customer already holds. Applications call one endpoint with an INFRO gateway key; a request names either a model, which resolves over the customer's own connections with failover between them, or a route. Routes take priority or weighted targets, up to 20 if-rules on request metadata, feature, end user, key, project, prompt size, streaming, images, tools or time of day, a prompt classifier backed by Jev or another decision model the customer connects, and failure rules per failure kind. A route cannot reach a model the organization's allowlist forbids, and a budget refuses a request before any provider is called.
Every request records the route, the target position that served it, the connection and each failed attempt, so the measurements in this guide come from the request log rather than from a separate pipeline. Provider usage is billed by the customer's own providers; INFRO charges a subscription for the control plane and takes no share of model spend. The quickstart connects a first provider and sends a request through a route.
What INFRO does not do: it does not restart a streamed response on another target after the first event has been sent, and it does not choose a different model during failover unless the route or the request's fallbacks list names one.