Failover & fallbacks

Providers have bad minutes. INFRO absorbs them with two layers of resilience: connection failover, which tries the same model on another of your own connections, and model fallbacks, an ordered list of alternate models you set per request. Both operate strictly before the first token.

Connection failover is same-model and happens whenever you have more than one eligible connection (or a named route with more than one target). Fallbacks are cross-model and opt-in via the fallbacks array. Either way, the response names the model, provider and connection that served the request and the estimated provider cost.

Connection failover

When you name a model directly, INFRO tries every enabled, healthy connection in your organization that serves that model's vendor and matches the key's environment, in the order the connections were created. When you name a route, it tries the route's targets in priority or weighted order. If a connection fails before the first token, INFRO moves to the next one. Your client sees only a slightly higher time to first token. With a single connection there is nothing to fail over to.

Every upstream failure before the first token moves to the next connection: a provider 5xx, timeout, 429, rejected credential (401/403), 404, or even a 400/422 — providers differ on which optional parameters they accept. A provider 429 does not count against the connection's health. routing.regions and routing.policy remain accepted legacy request fields, but they do not constrain the locations of customer-owned connections or set their order.

INFRO's own refusals — invalid request body, 401 for your INFRO key, 402 budget or subscription, 403 allowlist, 404 unknown model — are decided before any provider is called and return immediately without failover.

Model fallbacks

Fallbacks cover the case where an entire model is unavailable — every connection serving it is erroring or rate-limited. Pass a fallbacks array of catalog model IDs and INFRO tries them in order, each over your own connections for that model's vendor, with the same connection failover as the primary. Fallbacks apply only when model names a model; when it names a route, the route's own targets are used and fallbacks is ignored.

fallbacksstring[]

Ordered list of up to five catalog model IDs to try once every connection for the primary model has failed before the first token. An ID that is not a catalog model (and not a bare model name that resolves as model would) is refused up front with 400 invalid_request_error, code unknown_fallback_model, naming each offending entry — for example `fallbacks[1]: foo/bar is not a model or route this organization can call.; so is a route slug, because fallbacks name models. A known model is skipped, without an error, when it is a different modality, is not permitted by your model allowlists, or has no connection that can serve it right now. Ignored when model names a route, though an unknown ID is still refused. Top-level request body field — pass it via extra_body` in the OpenAI Python SDK, or directly in the request body in TypeScript.

curl https://api.infro.io/v1/chat/completions \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "moonshot/kimi-k2.6",
    "messages": [
      {"role": "user", "content": "Summarize this changelog in three bullets."}
    ],
    "fallbacks": ["zai/glm-5.2"]
  }'

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
)

response = client.chat.completions.create(
    model="moonshot/kimi-k2.6",
    messages=[
        {"role": "user", "content": "Summarize this changelog in three bullets."}
    ],
    extra_body={"fallbacks": ["zai/glm-5.2"]},
)

print(response.model)  # the model that actually served

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
});

const response = await client.chat.completions.create({
  model: "moonshot/kimi-k2.6",
  messages: [
    { role: "user", content: "Summarize this changelog in three bullets." },
  ],
  // INFRO extension — passed through in the request body
  // @ts-expect-error
  fallbacks: ["zai/glm-5.2"],
});

console.log(response.model); // the model that actually served

Request lifecycle

1. INFRO resolves the primary model to your eligible connections (or a route to its targets) and tries the first one.
2. If that attempt fails before the first token, INFRO tries the next connection for the same model — except where another key of the same provider would fail the same way: a request the provider rejected as invalid (400/422) goes straight to a different provider, and a timeout on a model INFRO spreads over your keys (no route) is not waited out again on another key of the same provider. A route's own targets and failure rules are kept as you set them.
3. When every connection for the primary model has failed, INFRO moves to the first usable entry in fallbacks and repeats the same connection failover — then the second entry, and so on.
4. When the chain is exhausted, the request fails with the most useful error: 403 if a provider's content moderation refused it; 400 with the provider's own message if every provider rejected the request as invalid; 429 provider_rate_limited with the provider's Retry-After if a provider rate-limited it; 503 model_not_served if every connection tried answered that it cannot serve the model, naming each one and what its provider said, for example The connection tried failed: LaoZhang key 1 (this key's group on the provider has no channel for the model (HTTP 503): "Current group default has no available channels for model …"). — give the key access on the provider's side, or connect another provider; 408 if every attempt timed out; otherwise 502 upstream_error, naming each connection that was tried with the provider's status and its own error message. If nothing was eligible to try, it is 503 no_available_provider. A key that answered it cannot serve a model is tried after your other keys for that model for the next fifteen minutes.

408, 429, 502 and 503 are retryable — use exponential backoff with jitter, and honor Retry-After on a 429. A 400 or 403 will fail the same way again. A 503 with code no_provider_connected means no connection serves the model's vendor, and model_not_served that your connections' keys have no access to the model; fix the connection rather than retrying. See Errors for the full retry matrix.

Reading the response

The response never hides what happened. Three fields report the truth:

modelstring

The model that actually served the request. With fallbacks configured, compare it against the model you asked for to detect a fallback serve.

providerstring

The provider kind of your connection that served the request, after any failover.

connectionstring

The ID of your connection that served the request.

routestring

The chain position that served: primary, then standby_a, standby_b, and so on. route_name carries the named route's slug when you called one.

usage.costnumber | null

An estimate of what your provider will bill, at the served model's rate (your custom rate if set, else the vendor's list rate). null when the model is unpriced. usage.price_source says where the rate came from and is present only when a cost was computed.

{
  "id": "req_8fJq3k",
  "object": "chat.completion",
  "model": "zai/glm-5.2",
  "route": "primary",
  "route_name": null,
  "provider": "together",
  "connection": "conn_7Qm2",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "- ..." },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 214,
    "completion_tokens": 96,
    "total_tokens": 310,
    "cost": 0.00044,
    "price_source": "list"
  }
}

Streaming

With stream: true, both layers apply in full before the first token — the only symptom of a failover chain is a longer wait for the first chunk. Once tokens flow, INFRO cannot restart your request on a different provider without silently replaying content, so it doesn't.

If a provider drops mid-stream, the stream surfaces an error chunk instead of data: [DONE]. Treat it as retryable: re-issue the request, and the full failover and fallback chain applies fresh. If you render partial output to users, be ready to discard it on retry. Chunk format and stream_options are covered in Streaming.

Choosing fallback models

A fallback only helps if its response is one your application can actually use. Pick fallbacks the way you would pick a substitute, not just a survivor:

• Match capabilities. If the primary route uses tool calling, structured outputs, or vision, every fallback must support the same features. The catalog does not record capabilities per model yet (capabilities in GET /v1/models is empty), so check the vendor's documentation and test the fallback directly. A fallback that can't emit tool_calls breaks your route in a new way instead of saving it.
• Match context length. A fallback with a smaller window turns a request that fits today into a 400 during an outage — exactly when you can least afford it.
• Match tier and output style. moonshot/kimi-k2.6 falling back to zai/glm-5.2 works because they are peers: similar quality, similar formats, both tool-capable. Pairing a frontier model with a budget one is a different decision entirely.
• Keep the chain short. One or two entries. Each exhausted model adds latency before a real outage surfaces, and a long chain usually means the route's requirements aren't pinned down.

Fallbacks apply only to requests that name a model; a request that names a route uses that route's own targets and ignores fallbacks — but an entry that is not a model at all is refused with 400 unknown_fallback_model on both kinds of request, so a typo never passes for a working fallback. A fallback is still subject to your model allowlists — it never routes around policy. See Routing for how the two interact, and Chat completions for the full request schema.