What changed, and what the announcement does not say
On November 16 OpenRouter announced that chat models on Hugging Face Inference Providers now run through OpenRouter's routing, with provider choice based on live price, speed and reliability.1 It says automatic failover applies: if a provider returns an error, the request is retried on the next one inside the same call.1 It lists 160 models from 38 labs, and model suffixes such as :fastest, :cheapest or a provider name tell the router what to prioritize.1
That is a routing layer you did not choose, and it sits behind an endpoint you may already call. The announcement does not describe the wire path, which headers come back, or whether a call made with a Hugging Face token is visible in an OpenRouter account. This article treats those as open questions and builds a test around them. The chain it examines is your application, then your gateway, then an endpoint that routes (OpenRouter itself, or Hugging Face Inference Providers running on OpenRouter's routing), then a model provider.
The same issue applies if your gateway points at OpenRouter directly. Two routers each decide where a request goes, and neither knows what the other will do.
Who owns a retry when two routers can both retry
The inner router's behavior is documented. By default OpenRouter ranks providers by measured price, throughput, latency and reliability, and if a try fails the request goes to the next highest-ranked provider.2 allow_fallbacks defaults to true.3 With a models array, any error can trigger the next model, including context-length errors, moderation flags, rate limits and downtime.4
Your gateway has its own policy. INFRO's, for example, moves to the next connection on a provider 5xx, timeout, 429, rejected credential, 404 or even a 400/422 before the first token.5 Most gateways are similarly generous. Stack the two and the failure your client sees is the last failure after both layers gave up.
| Layer | Typical default | What it can do to the other |
|---|---|---|
| Outer gateway | Retry on timeout, 5xx, 429 | Re-sends a request the inner router is still working on |
| Inner router | Try next provider on error | Hides the failure from the gateway, so its health check sees a slow success |
| Both | Independent budgets | Attempts multiply, and so does the time spent in failed ones |
The fix is a decision, not a setting. Choose the layer that owns failover for this model, and make the other behave as a pass-through: one attempt, or a single retry only against a different upstream. The inner router has better live data on its providers; your gateway has better knowledge of your own contracts and policy. Which one wins depends on whether you need to restrict providers (for data policy) or maximize availability.
Timeouts: the outer budget must exceed the inner one
OpenRouter's own experiment gives scale for the inner layer. In its switchback test over a week of open-source model traffic, 10.4% of requests under the old router needed a retry against 6.9% under the new one, and time spent in failed attempts averaged 242 ms per request under the new router.2 Those are averages for OpenRouter's sample, not your workload. The tail matters more: whole-request p95 was 34.9 s under the new router.2 A gateway timeout set from median latency will cut off requests that the inner router would have completed.
Set the budget in this order:
- Measure the inner endpoint's p95 and p99 time to first token and whole-request time for your own prompts.
- Set the outer timeout above the inner p99 plus one failover step.
- Set the outer retry count so that attempts times the outer timeout fits inside the user-facing deadline.
- For streaming, decide what happens after the first token. INFRO's failover operates only before the first token;5 how the inner router behaves mid-stream is not covered by the pages read here, so test it.
A retry on timeout deserves particular care. If the outer layer abandons a call that the inner provider already started, the provider may still bill the work. See what failover does to your bill and hedging and retry budgets.
Health signals and cache pinning disagree across layers
Two behaviors in the inner router interact with an outer one.
First, the inner router sends every provider a floor of trickle traffic so it can notice recovery, and backs off providers that fail or rate limit.2 An outer health check that only sees successes will report the chain healthy while the inner layer burns attempts. Count inner retries, not just outer errors.
Second, a conversation stays on the provider holding its prompt cache; only the first request has a provider chosen algorithmically, and if that provider becomes unavailable the conversation falls back automatically.2 The Hugging Face announcement repeats the pinning.1 If your gateway spreads requests of one conversation across several upstreams, or fails over mid-conversation, you restart the cache on a new provider and may change cost per request. Keep a conversation on one upstream at the outer layer and let the inner layer decide the provider.
Explicit controls change who is deciding. provider.order bypasses the new routing algorithm and routes in the order you give; only and ignore filter the candidates but the algorithm still picks among those left; an explicit sort replaces the algorithm.2 If you pin one provider, check whether failover to others remains on, since allow_fallbacks defaults to true.3 Whether a provider-name suffix on the Hugging Face path behaves like order with fallbacks off is not stated in the announcement. Test it.
Attribution: which hop's log can name the provider
When something is slow, you need to know which provider served the request. Each hop records a different thing.
- OpenRouter's fallback docs say requests are priced using the model ultimately used, returned in the
modelattribute of the response body.4 That names the model, and the pages read here do not show a provider field on the Hugging Face path. - The announcement says each model page has a Providers tab with live price, speed and uptime, and that you can see spend by provider.1 That is a dashboard on the router's side, not a field in your gateway's request log.
- INFRO's request log records the model, the selected provider connection, outcome, timings and usage.6 Its failover docs say the response names the model, provider and connection that served the request.5 "Provider" there means the upstream you connected. If that upstream is a router, the real serving provider is one hop further and is not in the gateway record unless the router returns it.
So a gateway log can honestly say "served by upstream X" while hiding that X retried twice on its own. To close the gap, record the router's response headers and body fields verbatim next to your request id, and join them to the router-side activity export by time and key. If neither names the provider on your path, write that down as a known limit rather than inferring one.
Data path and disclosure
Each hop is a processor of prompts and responses, and your security reviewer will ask for the list. Draw it explicitly: client, gateway, router, model provider, plus any copies kept on the side.
Three points need confirming, not assuming:
- Provider data policies. OpenRouter reflects each provider's policies in structured data and lets you restrict requests with
data_collection,zdrand similar fields; it also notes it has no routing rules that change on retention policy by default, so you opt in by ignoring providers that do not meet your requirements.73 Whether those request fields apply to a call arriving through a Hugging Face token is not stated. The announcement points to Hugging Face settings for choosing which providers may serve you.1 Check both and use the stricter. - Regional processing. OpenRouter documents in-region routing for the EU and US on Business and Enterprise plans, through separate hostnames.7 Nothing read here says that applies on the Hugging Face path. Do not claim regional processing for that traffic without written confirmation.
- Extra copies. OpenRouter can send a trace of each request to a Hugging Face Storage Bucket you own, holding the input messages, the output, token counts and cost, with media left out; the feature is in beta.1 The bucket is billed on data stored.8 That is a second store of prompts and completions beside whatever your gateway keeps. If your gateway's payload logging is off by default and retention-limited,6 a broadcast destination can undo that choice. Decide deliberately.
For retention policy wording, see zero data retention across providers.
A test to run before you trust the chain
Run this against a non-production key, with a model served by more than one provider. The goal is to measure attempts, time and attribution when something fails, because the happy path will look fine.
- Baseline. Send 200 identical short requests through the full chain. Record status, total latency, time to first token, the
modelfield and every response header. - Count hops. Tag each request with a unique id header. Compare the count at the gateway, at the router's activity view, and at any provider-side usage you can read. Any mismatch is hidden retries.
- Break the outer hop. Point the gateway's upstream at a blackholed address for one connection. Confirm the gateway fails over once, to the intended alternative, and logs it.
- Break the inner hop. Pin a provider you know to be failing, or send a request the first provider will reject, such as an unsupported parameter. Check whether the inner router moves on and whether the outer layer sees a retry, an error or a slow success.
- Timeout race. Set the outer timeout below the inner failover time. Count the upstream attempts per client request. Then raise it above and confirm the count drops.
- Stream cut. Kill a stream after the first chunk and note what each layer does. Do not assume a retry restarts it.
- Attribution audit. For ten failures, try to name the serving provider from your own logs alone. Record how many you can.
import time, uuid, json, httpx
URL = "https://gateway.example.com/v1/chat/completions"
KEY = "GATEWAY_KEY"
def probe(model: str, n: int = 200):
rows = []
with httpx.Client(timeout=60) as c:
for _ in range(n):
rid = str(uuid.uuid4())
t0 = time.monotonic()
r = c.post(URL, headers={"Authorization": f"Bearer {KEY}",
"x-probe-id": rid},
json={"model": model, "max_tokens": 16,
"messages": [{"role": "user", "content": "ping"}]})
rows.append({
"probe_id": rid,
"status": r.status_code,
"seconds": round(time.monotonic() - t0, 3),
"model": (r.json().get("model") if r.status_code == 200 else None),
"headers": dict(r.headers),
})
return rows
print(json.dumps(probe("your-model-id")[:3], indent=2))Pass criteria to write down in advance: upstream attempts per client request stay at or below your budget under fault injection; the outer timeout exceeds the inner failover time; every failure can be attributed to at least a hop; and the data-path list matches what your reviewer approved.
Where a gateway helps, and where it cannot see
A gateway with its own routing can own the outer decision. INFRO's routes use priority or weighted targets over your own provider connections with health checks, retries and failover, and its request log records the connection that served each request and its fallback attempts.56 That gives you the outer half of the evidence. It cannot see inside a router you call as an upstream beyond what that router returns, so the inner-hop checks above remain yours to run.
