Skip to content

Routing and reliabilityMethodology

What uptime can a fallback chain promise? A composite SLO method

A fallback chain does not multiply provider SLAs into five nines. This method builds the target from measured per-provider error rates, a correlation term, and the gateway and client terms in series, then turns it into an error budget you can commit to.

By INFRO Editorial

Published

9 min read

Luminous routing paths connecting one application node to many AI model endpoints

In short

Compute the chain's availability from measured per-provider failure rates, add a common-mode term for failures the providers share, multiply by the gateway and client terms in series, and commit below the result. Provider SLAs define availability narrowly and pay only a bill reduction, so they are inputs, not the target. 12

Key takeaways

  • A provider SLA is a narrow, region- or project-scoped definition with a bill-reduction remedy; it is an input to your SLO, never the SLO itself.
  • Chain availability is 1 minus the product of failure rates only if failures are independent; add a common-mode term estimated from your own history, and it will usually dominate.
  • Multiply the chain by the terms in series: your gateway, client timeouts, failures after the first token, and refusals such as budgets and quotas.
  • Count 429s, timeouts and cut-off streams as bad in your SLI even when provider SLAs exclude them.
  • Commit to a number below the computed one, express it as an error budget in minutes and requests, and re-estimate every quarter.

Why provider SLAs do not add up to your SLO

The tempting calculation is to take two providers at 99.9% each and write down 99.9999% for the chain. It fails for three reasons: the SLAs measure something narrower than your users do, their remedy is not an availability guarantee, and the failures are not independent.

Read what the definitions actually count. Amazon's Bedrock SLA computes availability per 5-minute interval as the share of requests that do not fail with an Error, and defines an Error as a request that returns a 500 error code; an interval with no requests is assumed 100% available.1 A 429, a 503, a timeout or a stream that stops halfway is not in that definition. It is also scoped per AWS region and per account, and it excludes, among other things, problems caused by your own configuration and underlying software that causes repeated model crashes.1 Google's Vertex AI SLA defines Downtime as more than a 5% error rate over a minute, counting only HTTP 500 and 503 responses, measured server side, and it excludes errors that result from quotas.2 The covered-service table on that page lists training, deployment, batch prediction, AutoML and custom model online prediction; check whether the endpoint you call is a covered service before you use any figure from it.2

The remedy is also not uptime. Bedrock's tiers are 10%, 25% and 100% of the affected region's charges for the month, applied against future payments and claimed through a support case with your own request logs.1 Vertex's tiers are 10%, 25% and 50%, capped at 50% of the month's amount for the covered service, and you must request them within 30 days.2 Neither pays for your outage. So treat the SLA as a statement about what the provider will measure, and build your target from what you measure.

Define the SLI the user would define

An SLO needs a service-level indicator first. For an AI feature, write it from the caller's side:

  • Good event: the request returned a usable response within your latency threshold. Include a time-to-first-token or total-time bound, because a 90-second answer to a chat turn is down for the user.
  • Bad events: any 5xx, any timeout, a 429 that survived the whole chain, a stream cut off after the first token, and a refusal that stops the request (budget, allowlist, guardrail).
  • Not counted: requests your own validation rejects as malformed, which are a bug in the caller, not an outage.

Counting 429s and cut-off streams is the important departure from the provider SLAs above. A capacity shortage that returns 429 is invisible to a 500-only definition and fully visible to your user.

It also matters where failover can act. INFRO's documentation states that both connection failover and model fallbacks operate strictly before the first token, and that every upstream failure before the first token (5xx, timeout, 429, rejected credential, even a 400 or 422) moves on to the next connection.3 Failures after the first token are outside what a chain can rescue, so they form their own term in the calculation. See streaming failure modes for how often that term matters for your traffic.

Gateway refusals are a separate class. INFRO's own refusals, such as an invalid body, a budget, an allowlist or an unknown model, are decided before any provider is called and are not failed over.3 Decide up front which of them you count against the SLO. A budget refusal is working as designed, but a user who is refused still did not get an answer; many teams exclude it from the SLI and track it as a separate rate.

The chain term: independent first, then correlated

Take a feature with two targets, A then B. Measure each one's per-attempt failure rate over a long window, using your own request records: u_A and u_B. If failures were independent, a request fails only when both fail:

U_indep = u_A × u_B

That number is a ceiling, not an estimate. To make it honest, add a common-mode term using a beta factor β: the fraction of the more reliable provider's failures that happen together with the other's.

U_chain = (1 − β) × u_A × u_B + β × min(u_A, u_B)

Estimate β from history, not from hope. Using the Vertex definition as a convenient yardstick, mark a minute as "down" for a provider when its error rate in your traffic exceeds 5%. Then β ≈ (minutes where both are down) / (minutes where the more reliable one is down). It is a modeling choice, not a standard, and with few incidents it is noisy, so keep the raw counts beside the estimate.

composite_slo.py
def chain_unavailability(u_a: float, u_b: float, beta: float) -> float:
    """Two-target chain; beta = share of the smaller failure rate that is common-mode."""
    return (1 - beta) * u_a * u_b + beta * min(u_a, u_b)

def composite(terms: list[float]) -> float:
    """Series terms (availabilities): gateway, chain, post-first-token, ..."""
    out = 1.0
    for a in terms:
        out *= a
    return out

chain = 1 - chain_unavailability(0.005, 0.007, beta=0.1)   # illustrative inputs
gateway = 0.999        # placeholder: use your own gateway's measured availability
mid_stream = 0.999     # placeholder: measured share of streams cut after first token
print(f"{composite([gateway, chain, mid_stream]):.5f}")    # ~0.99747

For three or more targets, extend the same idea: the chain fails when all targets fail, and the common-mode term is the fraction of time that all of them are down together, which you can count directly from minute-level history instead of modeling.

Series terms: everything the chain sits inside

The chain is one block in a series. Multiply its availability by every component that every request must pass through:

  1. The gateway or proxy in front of the providers. If you run INFRO, its published position is a 99.9% target per calendar month on Business and enterprise agreements, measured by its own minute checks, where a failed check or more than two minutes with no check counts as downtime; on a published plan it is a target and brings no refund.4 The status page is explicit that this covers the gateway, not your provider accounts, and that the record is kept inside the vendor's own infrastructure.4 Use the gateway's measured figure, not the target, once you have some history, and note the no-refund caveat: it is a number to plan with, not insurance.
  2. Your own client and network path, including DNS, egress and timeouts.
  3. Post-first-token failures, from the previous section.
  4. Anything that refuses before a provider is called: spend caps, key controls, allowlists, guardrails. INFRO's error reference (opens in a new tab) lists a gateway_unavailable 503 among the codes that surface only when no connection can serve the request.5

In the worked example, a 99.9% gateway, the 99.947% chain and a 99.9% post-first-token term multiply to about 99.75%. The gateway term is larger than the whole chain's failure budget. Adding a third provider does almost nothing to that number; hardening the layers in front of the chain does far more. Whether the application should fail open to a direct provider call when the gateway is down is a separate design choice, covered in gateway down: fail open or fail closed.

Correlated failures: where β comes from

The common-mode term is not abstract. These are the places failures line up:

  • A shared cloud or region. The same model family may be offered by its maker and by a hyperscaler, and two hyperscalers may both be fronted by the same CDN, identity service or DNS. SLAs are scoped per region or per project,12 so a regional event is one outage in two contracts. Check where each of your connections is actually hosted before calling them independent.
  • Same-model failover. Routing the same model through two hosts removes host failures but keeps model-level faults: a bad snapshot, a safety-filter regression or a capacity shortage on that model.
  • Shared quota or account. Two connections that are the same organization's keys share its rate limit. INFRO's connection failover spreads requests across a provider's active keys and tries standbys last,3 but if the limit is per organization, more keys add no capacity.
  • Shared demand spikes. When a popular model launches or a competitor has an incident, traffic moves and several providers throttle at once.
  • Your own dependencies. The same egress NAT, secret store or deploy pipeline sits under every target.

A cross-model fallback has a different weakness. It decorrelates model-level faults, but it changes answers, so "available" now depends on whether the fallback is good enough for that feature; test it with fallback equivalence before counting its successes as good events.

Measure the inputs from your own records

You need per-attempt outcomes, not just final outcomes: which target was tried, in what order, and how it failed. INFRO's request records name the model, the connection that served each request and the route,6 and GET /v1/requests and a CSV export of up to 50,000 rows per file are available, bounded by your plan's queryable history of 7 to 365 days.7 OpenTelemetry spans can also be pushed to your own collector, but delivery is best effort with one retry and dropped spans on persistent failure, so the documentation says not to use that channel as the sole accounting record.7 For an SLO, export the request records on a schedule and keep them yourself, because a 7-day window will not cover a 90-day estimate.

A minimal estimator over per-minute data (column names are illustrative; map them from your export):

estimate_beta.py
import pandas as pd

# minute, provider, attempts, failures  (aggregated from your exported request records)
df = pd.read_csv("attempts_by_minute.csv", parse_dates=["minute"])
df["down"] = (df.failures / df.attempts) > 0.05      # Vertex-style 5% minute rule, as a yardstick

wide = df.pivot(index="minute", columns="provider", values="down").fillna(False)
a_down, b_down = wide["provider_a"], wide["provider_b"]

both = (a_down & b_down).sum()
rarer = min(a_down.sum(), b_down.sum())
print("minutes both down:", both, "of", rarer, "-> beta ~", both / rarer if rarer else None)

Report the counts with the ratio. A β of 0.1 from 3 incident minutes is a guess; from 300 it is a measurement. Re-estimate it quarterly and after any change in where targets are hosted.

From composite to commitment and error budget

Do not commit to the computed number. Subtract a margin for what the model cannot see: unmeasured correlation, a future provider with a worse year, and the variance of a short window. A common practice is to commit to a target whose error budget is at least about twice what the model predicts you will spend. In the example, the model predicts about 99.75%, so a 99.5% commitment leaves roughly half of its budget unspent.

Then state the budget in units people can act on. Over a 30-day month (43,200 minutes):

TargetBudget (fraction)Budget in minutes of full outage
99.5%0.5%216
99.9%0.1%43.2
99.95%0.05%21.6
Error budget over a 30-day month; request-based SLOs spend the same fraction of requests instead of minutes.

Because AI traffic is bursty and uneven, express the budget as a fraction of requests, not minutes, so a quiet night does not hide a bad hour. Alert on burn rate: page when a short window and a longer window are both consuming budget much faster than the month allows, and open a ticket on a slow burn. When the budget is spent, the policy is the usual one: freeze risky changes to routes, models and prompts until it recovers.

Finally, write the assumptions on the SLO page itself: the SLI definition, the window used to estimate each term, the β and how it was obtained, and which refusals are excluded. When any of those change, the number changes.

Building the chain and the record in INFRO

INFRO can supply the mechanics this method needs: routes (opens in a new tab) with priority or weighted targets over your own connections, which still fall through to the rest when the first pick fails,6 failover and fallbacks (opens in a new tab) before the first token, and per-request records with the connection that served each attempt for the estimates above. It does not make the chain's failures independent, and its own availability is one of the series terms to include.

Questions readers ask

Should the SLO be per model, per provider or per feature?
Per feature. The user experiences the feature, not the provider. Compute a composite for each feature from the chain it actually uses, since a feature with a cross-model fallback and one with same-model failover have different correlation terms.
How much history do I need before committing?
Ninety days is a workable minimum, because a 30-day window may hold only one or two provider incidents and the common-mode estimate rests on those events. Until you have it, commit to a lower number and say it is provisional.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Health-aware routing

    Available with a plan

    Named routes use priority or weighted targets over your own connections, with health checks, retries and failover.

    Read the documentation for Health-aware routing
  • Automatic failover

    Available with a plan

    A route that times out or 5xxs before the response starts is retried elsewhere without your client seeing it.

    Read the documentation for Automatic failover
  • Request traces and payload logs

    Available with a plan

    Per-request timing detail, with prompt and completion logging off by default and a retention window you choose.

    Read the documentation for Request traces and payload logs
  • OpenTelemetry export

    Available with a plan

    Spans pushed to your own collector from the request path, with the endpoint and headers set in the console. Included on every plan.

    Read the documentation for OpenTelemetry export
  • 99.9% uptime commitment

    Available with a plan

    The gateway available 99.9% of each calendar month on Business and enterprise agreements, measured by its own minute checks and published at /status. A target: no refund applies on a published plan.

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Amazon Bedrock Service Level Agreement (opens in a new tab) — Amazon Web Services. Accessed Nov 15, 2026.
  2. 2.Vertex AI Platform Service Level Agreement (SLA) (opens in a new tab) — Google Cloud. Accessed Nov 15, 2026.
  3. 3.Failover & fallbacks (opens in a new tab) — INFRO Docs. Accessed Nov 15, 2026.
  4. 4.Gateway status and uptime (opens in a new tab) — INFRO. Accessed Nov 15, 2026.
  5. 5.Errors — API reference (opens in a new tab) — INFRO Docs. Accessed Nov 15, 2026.
  6. 6.Routes (opens in a new tab) — INFRO Docs. Accessed Nov 15, 2026.
  7. 7.Exports & OpenTelemetry (opens in a new tab) — INFRO Docs. Accessed Nov 15, 2026.

About this article

Read on 2026-11-15: the Amazon Bedrock SLA, the Vertex AI Platform SLA, INFRO's fallbacks, routes, errors and exports docs, and INFRO's status page. The Azure OpenAI SLA could not be fetched, and the OpenTelemetry GenAI page only pointed to a moved repository, so neither is quoted. All availability figures in the worked example are illustrative.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.