# Failover & fallbacks

> How INFRO keeps requests flowing: automatic provider failover, ordered model fallbacks, streaming behavior, and how to pick safe fallback models.

Source: https://infro.io/docs/features/fallbacks

> **Note:** INFRO is in private early access. Running these examples requires an early-access account, an active trial or subscription entitlement, and your own provider connection. Self-serve purchasing is not generally open. See [Early access](https://infro.io/early-access).

Providers have bad minutes. INFRO absorbs them with two layers of resilience: **connection failover**, which tries the same model on another of your own connections, and **model fallbacks**, an ordered list of alternate models you set per request. Both operate strictly before the first token.

Connection failover is same-model and happens whenever you have more than one eligible connection (or a named route with more than one target). Fallbacks are cross-model and opt-in via the `fallbacks` array. Either way, the response names the model, provider and connection that served the request and the estimated provider cost.

## Connection failover

When you name a model directly, INFRO tries every enabled, healthy connection in your organization that serves that model's vendor and matches the key's environment, in the order the connections were created. When you name a [route](https://infro.io/docs/routes), it tries the route's targets in priority or weighted order. If a connection fails before the first token, INFRO moves to the next one. Your client sees only a slightly higher time to first token. With a single connection there is nothing to fail over to.

Every upstream failure before the first token moves to the next connection: a provider 5xx, timeout, 429, rejected credential (401/403), 404, or even a 400/422 — providers differ on which optional parameters they accept. A provider 429 does not count against the connection's health. `routing.regions` and `routing.policy` remain accepted legacy request fields, but they do not constrain the locations of customer-owned connections or set their order.

INFRO's own refusals — invalid request body, 401 for your INFRO key, 402 budget or subscription, 403 allowlist, 404 unknown model — are decided before any provider is called and return immediately without failover.

## Model fallbacks

Fallbacks cover the case where an entire model is unavailable — every connection serving it is erroring or rate-limited. Pass a `fallbacks` array of catalog model IDs and INFRO tries them in order, each over your own connections for that model's vendor, with the same connection failover as the primary. Fallbacks apply only when `model` names a model; when it names a route, the route's own targets are used and `fallbacks` is ignored.

- `fallbacks` (string[]) — Ordered list of up to five catalog model IDs to try once every connection for the primary model has failed before the first token. An ID that is not a catalog model (and not a bare model name that resolves as `model` would) is refused up front with `400 invalid_request_error`, code `unknown_fallback_model`, naming each offending entry — for example ``fallbacks[1]: `foo/bar` is not a model or route this organization can call.``; so is a route slug, because fallbacks name models. A known model is skipped, without an error, when it is a different modality, is not permitted by your model allowlists, or has no connection that can serve it right now. Ignored when `model` names a route, though an unknown ID is still refused. Top-level request body field — pass it via `extra_body` in the OpenAI Python SDK, or directly in the request body in TypeScript.

**curl**

```bash
curl https://api.infro.io/v1/chat/completions \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshot/kimi-k2.6",
    "messages": [
      {"role": "user", "content": "Summarize this changelog in three bullets."}
    ],
    "fallbacks": ["zai/glm-5.2"]
  }'
```

**Python**

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
)

response = client.chat.completions.create(
    model="moonshot/kimi-k2.6",
    messages=[
        {"role": "user", "content": "Summarize this changelog in three bullets."}
    ],
    extra_body={"fallbacks": ["zai/glm-5.2"]},
)

print(response.model)  # the model that actually served
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
});

const response = await client.chat.completions.create({
  model: "moonshot/kimi-k2.6",
  messages: [
    { role: "user", content: "Summarize this changelog in three bullets." },
  ],
  // INFRO extension — passed through in the request body
  // @ts-expect-error
  fallbacks: ["zai/glm-5.2"],
});

console.log(response.model); // the model that actually served
```

## Request lifecycle

1. INFRO resolves the primary model to your eligible connections (or a route to its targets) and tries the first one.
2. If that attempt fails before the first token, INFRO tries the next connection for the same model — except where another key of the same provider would fail the same way: a request the provider rejected as invalid (`400`/`422`) goes straight to a different provider, and a timeout on a model INFRO spreads over your keys (no [route](https://infro.io/docs/routes)) is not waited out again on another key of the same provider. A route's own targets and failure rules are kept as you set them.
3. When every connection for the primary model has failed, INFRO moves to the first usable entry in `fallbacks` and repeats the same connection failover — then the second entry, and so on.
4. When the chain is exhausted, the request fails with the most useful error: **403** if a provider's content moderation refused it; **400** with the provider's own message if every provider rejected the request as invalid; **429** `provider_rate_limited` with the provider's `Retry-After` if a provider rate-limited it; **503** `model_not_served` if every connection tried answered that it cannot serve the model, naming each one and what its provider said, for example `The connection tried failed: LaoZhang key 1 (this key's group on the provider has no channel for the model (HTTP 503): "Current group default has no available channels for model …").` — give the key access on the provider's side, or connect another provider; **408** if every attempt timed out; otherwise **502** `upstream_error`, naming each connection that was tried with the provider's status and its own error message. If nothing was eligible to try, it is **503** `no_available_provider`. A key that answered it cannot serve a model is tried after your other keys for that model for the next fifteen minutes.

408, 429, 502 and 503 are retryable — use exponential backoff with jitter, and honor `Retry-After` on a 429. A 400 or 403 will fail the same way again. A 503 with code `no_provider_connected` means no connection serves the model's vendor, and `model_not_served` that your connections' keys have no access to the model; fix the connection rather than retrying. See [Errors](https://infro.io/docs/api/errors) for the full retry matrix.

## Reading the response

The response never hides what happened. Three fields report the truth:

- `model` (string) — The model that actually served the request. With fallbacks configured, compare it against the model you asked for to detect a fallback serve.
- `provider` (string) — The provider kind of your connection that served the request, after any failover.
- `connection` (string) — The ID of your connection that served the request.
- `route` (string) — The chain position that served: `primary`, then `standby_a`, `standby_b`, and so on. `route_name` carries the named route's slug when you called one.
- `usage.cost` (number | null) — An estimate of what your provider will bill, at the served model's rate (your custom rate if set, else the vendor's list rate). `null` when the model is unpriced. `usage.price_source` says where the rate came from and is present only when a cost was computed.

```json
{
  "id": "req_8fJq3k",
  "object": "chat.completion",
  "model": "zai/glm-5.2",
  "route": "primary",
  "route_name": null,
  "provider": "together",
  "connection": "conn_7Qm2",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "- ..." },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 214,
    "completion_tokens": 96,
    "total_tokens": 310,
    "cost": 0.00044,
    "price_source": "list"
  }
}
```

> **Note:** Cost follows the served model, not the requested one. If `moonshot/kimi-k2.6` is down and `zai/glm-5.2` serves, your provider bills glm-5.2 rates and `usage.cost` estimates them. Vendor list rates for every model are listed at [/providers](https://infro.io/providers).

## Streaming

With `stream: true`, both layers apply in full before the first token — the only symptom of a failover chain is a longer wait for the first chunk. Once tokens flow, INFRO cannot restart your request on a different provider without silently replaying content, so it doesn't.

If a provider drops mid-stream, the stream surfaces an error chunk instead of `data: [DONE]`. Treat it as retryable: re-issue the request, and the full failover and fallback chain applies fresh. If you render partial output to users, be ready to discard it on retry. Chunk format and `stream_options` are covered in [Streaming](https://infro.io/docs/api/streaming).

## Choosing fallback models

A fallback only helps if its response is one your application can actually use. Pick fallbacks the way you would pick a substitute, not just a survivor:

- **Match capabilities.** If the primary route uses [tool calling](https://infro.io/docs/features/tool-calling), [structured outputs](https://infro.io/docs/features/structured-outputs), or vision, every fallback must support the same features. The catalog does not record capabilities per model yet (`capabilities` in `GET /v1/models` is empty), so check the vendor's documentation and test the fallback directly. A fallback that can't emit `tool_calls` breaks your route in a new way instead of saving it.
- **Match context length.** A fallback with a smaller window turns a request that fits today into a 400 during an outage — exactly when you can least afford it.
- **Match tier and output style.** `moonshot/kimi-k2.6` falling back to `zai/glm-5.2` works because they are peers: similar quality, similar formats, both tool-capable. Pairing a frontier model with a budget one is a different decision entirely.
- **Keep the chain short.** One or two entries. Each exhausted model adds latency before a real outage surfaces, and a long chain usually means the route's requirements aren't pinned down.

> **Warning:** For critical routes, beware the silent downgrade: with a much weaker fallback, requests keep succeeding, quality drops, and nothing errors. Log the response `model` field and alert when it differs from the primary. For some routes, a 502 you handle explicitly beats output you can't trust.

Fallbacks apply only to requests that name a model; a request that names a [route](https://infro.io/docs/routes) uses that route's own targets and ignores `fallbacks` — but an entry that is not a model at all is refused with `400` `unknown_fallback_model` on both kinds of request, so a typo never passes for a working fallback. A fallback is still subject to your model allowlists — it never routes around policy. See [Routing](https://infro.io/docs/features/routing) for how the two interact, and [Chat completions](https://infro.io/docs/api/chat-completions) for the full request schema.