LLM routing is the practice of choosing the best model or provider for each request at runtime. The strongest production policy combines capability rules, live health, latency, cost, and explicit fallbacks; it does not simply send every prompt to the most powerful model.
The INFRO view: INFRO centralizes routing decisions so teams can change providers and models without scattering vendor logic through the product.
Why routing belongs outside application code
Model choice changes faster than product code should. Prices move, providers degrade, context requirements vary, and a model that wins one evaluation may lose the next. If every call site contains vendor-specific branching, each change becomes a release and each outage becomes an incident response exercise.
A gateway turns model choice into infrastructure policy. The application describes what it needs; the routing layer decides where the request should run and records why that path was selected.
Five routing strategies that work
Most teams need a blend of strategies rather than one global rule.
- Capability routing: restrict requests to models that support the required context, tools, modality, or structured output.
- Cost routing: send high-volume, verifiable work to the lowest-cost model that clears an evaluation threshold.
- Latency routing: prefer the route with the best recent time-to-first-token for interactive experiences.
- Reliability routing: remove unhealthy providers from consideration and recover on an eligible standby.
- Escalation routing: start with an efficient model and escalate only low-confidence or failed work to a premium model.
The signals a router should use
Static price tables are only one input. A useful router also needs recent errors, timeouts, rate limits, response latency, model eligibility, remaining budget, and the application's quality policy. Each signal needs a clear freshness window so a temporary spike does not cause route flapping.
Keep hard constraints separate from preferences. A model without vision support is ineligible for an image request; a model that is ten percent slower is merely less preferred. That distinction makes policies explainable and easier to test.
How to validate a routing policy
Replay a representative evaluation set through every candidate route, then compare success rate, cost per successful task, p95 latency, and malformed-output rate. Shadow traffic is useful when privacy policy permits it, but an offline replay is enough to catch most routing mistakes before launch.
Finally, test failure deliberately: inject a timeout, a rate limit, and a malformed stream. The router is production-ready only when the application receives a predictable result and the trace explains the decision.
Where INFRO fits
INFRO gives each request one OpenAI-compatible text entry point, applies model access and routing policy, and records route, latency, usage, and cost in one trace. Eligible failures can move to a healthy standby without changing the client contract.
Use the model catalog to shortlist candidates, the pricing tools to model the economics, and the quickstart to run the same request across those candidates. INFRO is in private early access, so production eligibility and available routes should be confirmed during onboarding.
Frequently asked questions
What is the best LLM routing strategy?
There is no universal best strategy. Start with capability constraints, remove unhealthy routes, and rank the remaining candidates by the metric that matters for the workload—usually quality-adjusted cost or interactive latency.
Does LLM routing reduce quality?
It can if cost is the only signal. Quality-safe routing uses evaluation thresholds and sends only workloads a candidate model has already demonstrated it can handle.
Can INFRO route without changing my SDK?
For OpenAI-compatible text clients, adoption is primarily a base URL, API key, and model ID change. See the INFRO quickstart for the current contract.
