Topic
Routing and reliability
Routing policy, failover, retries, latency and incident response for AI traffic that has to keep working when a provider does not.
14 articles · Last updated
Start with
Routing & reliability
LLM routing strategies for reliable production AI
Routing decides which model and which provider account serve each request. This guide separates the two decisions, walks through five strategies and their failure modes, and shows how to test a policy before it carries production traffic.
GuideUpdated 10 min read
Routing & reliability
Automatic LLM failover: a production implementation guide
Failover is more than retrying a 500. This guide covers which failures to move, how many times, where streaming makes recovery impossible, when a different model is acceptable, and what operators need to see.
GuideUpdated 9 min read
Routing & reliability
LLM observability: what to measure in production
Logs are not observability. Production AI needs a record per request that ties together route, attempts, latency, tokens, estimated cost and outcome — and a privacy posture that does not depend on reading prompts.
GuideUpdated 11 min read
More in this topic
Routing & reliability
LLM routing rules: if-rules, failure rules, key spreading
A routing policy answers three questions for every request: which targets may serve it, which ones it should try and in what order, and what each kind of failure should do next. Write them down as data, not as retry loops.
Guide13 min read
Routing & reliability
September 3, 2026: provider incidents and failover
On 3 September 2026 three AI providers posted incidents within hours of each other. Their own status pages tell a more precise story than the headlines, and it is the story failover has to be designed for.
Analysis11 min read
Routing & reliability
Circuit breakers for LLM providers: throttling is not failure
The textbook circuit breaker counts errors. Point it at an LLM provider and the commonest error is a 429 from an account that is working fine. An analysis of what a breaker should count, what it should not, and what to do instead.
Analysis9 min read
Routing & reliability
Failover drills: test LLM provider failure before it happens
The failover path is the least-exercised code in most AI applications, and it runs for the first time during an outage. A method for drilling it on purpose: which faults to inject, a mock upstream that injects them, and the assertions that prove recovery happened.
Methodology10 min read
Routing & reliability
Provider rate limits compared: headers, tiers and 429s
Five providers, five rate-limit models. What each one counts, which headers tell you where you stand, which 429s clear on their own and which never will, and client code that treats them differently.
Comparison10 min read
Routing & reliability
Measuring time to first token and setting timeouts
Time to first token is the latency users feel in a streamed answer, and the number most teams measure loosely. A method for measuring it, reading it and turning it into timeouts that fail over before users give up.
Methodology9 min read
Routing & reliability
Streaming LLM failures: interrupted streams and recovery
Streaming moves errors from the status line into the body. This guide shows how streams actually end, why a closed connection is not a success, and client code that tells complete answers from interrupted ones.
Guide9 min read
Routing & reliability
LLM latency optimization: improve time to first useful answer
Average duration hides the experience. Measure queueing, time to first token, generation rate, tools, and retries—then optimize the stage users actually feel.
BriefUpdated 1 min read
Routing & reliability
The production LLM checklist: controls to verify before launch
Before real users arrive, verify the model, the failure path, the data path, key scope, tracing, budgets, rollback and ownership. This checklist turns readiness into evidence.
ChecklistUpdated 2 min read
Routing & reliability
LLM incident response: diagnose failures across providers
AI incidents cross application, gateway, provider, model, prompt, data, and tools. Triage quickly by separating availability, latency, quality, and spend.
BriefUpdated 1 min read
Routing & reliability
Quality-aware LLM routing: optimize cost without blind downgrades
Cost-aware routing becomes trustworthy only when quality is an explicit constraint. Evaluate by workload, route within thresholds, and escalate uncertainty.
BriefUpdated 1 min read