Why provider SLAs do not add up to your SLO
The tempting calculation is to take two providers at 99.9% each and write down 99.9999% for the chain. It fails for three reasons: the SLAs measure something narrower than your users do, their remedy is not an availability guarantee, and the failures are not independent.
Read what the definitions actually count. Amazon's Bedrock SLA computes availability per 5-minute interval as the share of requests that do not fail with an Error, and defines an Error as a request that returns a 500 error code; an interval with no requests is assumed 100% available.1 A 429, a 503, a timeout or a stream that stops halfway is not in that definition. It is also scoped per AWS region and per account, and it excludes, among other things, problems caused by your own configuration and underlying software that causes repeated model crashes.1 Google's Vertex AI SLA defines Downtime as more than a 5% error rate over a minute, counting only HTTP 500 and 503 responses, measured server side, and it excludes errors that result from quotas.2 The covered-service table on that page lists training, deployment, batch prediction, AutoML and custom model online prediction; check whether the endpoint you call is a covered service before you use any figure from it.2
The remedy is also not uptime. Bedrock's tiers are 10%, 25% and 100% of the affected region's charges for the month, applied against future payments and claimed through a support case with your own request logs.1 Vertex's tiers are 10%, 25% and 50%, capped at 50% of the month's amount for the covered service, and you must request them within 30 days.2 Neither pays for your outage. So treat the SLA as a statement about what the provider will measure, and build your target from what you measure.
Define the SLI the user would define
An SLO needs a service-level indicator first. For an AI feature, write it from the caller's side:
- Good event: the request returned a usable response within your latency threshold. Include a time-to-first-token or total-time bound, because a 90-second answer to a chat turn is down for the user.
- Bad events: any 5xx, any timeout, a 429 that survived the whole chain, a stream cut off after the first token, and a refusal that stops the request (budget, allowlist, guardrail).
- Not counted: requests your own validation rejects as malformed, which are a bug in the caller, not an outage.
Counting 429s and cut-off streams is the important departure from the provider SLAs above. A capacity shortage that returns 429 is invisible to a 500-only definition and fully visible to your user.
It also matters where failover can act. INFRO's documentation states that both connection failover and model fallbacks operate strictly before the first token, and that every upstream failure before the first token (5xx, timeout, 429, rejected credential, even a 400 or 422) moves on to the next connection.3 Failures after the first token are outside what a chain can rescue, so they form their own term in the calculation. See streaming failure modes for how often that term matters for your traffic.
Gateway refusals are a separate class. INFRO's own refusals, such as an invalid body, a budget, an allowlist or an unknown model, are decided before any provider is called and are not failed over.3 Decide up front which of them you count against the SLO. A budget refusal is working as designed, but a user who is refused still did not get an answer; many teams exclude it from the SLI and track it as a separate rate.
The chain term: independent first, then correlated
Take a feature with two targets, A then B. Measure each one's per-attempt failure rate over a long window, using your own request records: u_A and u_B. If failures were independent, a request fails only when both fail:
U_indep = u_A × u_B
That number is a ceiling, not an estimate. To make it honest, add a common-mode term using a beta factor β: the fraction of the more reliable provider's failures that happen together with the other's.
U_chain = (1 − β) × u_A × u_B + β × min(u_A, u_B)
Estimate β from history, not from hope. Using the Vertex definition as a convenient yardstick, mark a minute as "down" for a provider when its error rate in your traffic exceeds 5%. Then β ≈ (minutes where both are down) / (minutes where the more reliable one is down). It is a modeling choice, not a standard, and with few incidents it is noisy, so keep the raw counts beside the estimate.
def chain_unavailability(u_a: float, u_b: float, beta: float) -> float:
"""Two-target chain; beta = share of the smaller failure rate that is common-mode."""
return (1 - beta) * u_a * u_b + beta * min(u_a, u_b)
def composite(terms: list[float]) -> float:
"""Series terms (availabilities): gateway, chain, post-first-token, ..."""
out = 1.0
for a in terms:
out *= a
return out
chain = 1 - chain_unavailability(0.005, 0.007, beta=0.1) # illustrative inputs
gateway = 0.999 # placeholder: use your own gateway's measured availability
mid_stream = 0.999 # placeholder: measured share of streams cut after first token
print(f"{composite([gateway, chain, mid_stream]):.5f}") # ~0.99747For three or more targets, extend the same idea: the chain fails when all targets fail, and the common-mode term is the fraction of time that all of them are down together, which you can count directly from minute-level history instead of modeling.
Series terms: everything the chain sits inside
The chain is one block in a series. Multiply its availability by every component that every request must pass through:
- The gateway or proxy in front of the providers. If you run INFRO, its published position is a 99.9% target per calendar month on Business and enterprise agreements, measured by its own minute checks, where a failed check or more than two minutes with no check counts as downtime; on a published plan it is a target and brings no refund.4 The status page is explicit that this covers the gateway, not your provider accounts, and that the record is kept inside the vendor's own infrastructure.4 Use the gateway's measured figure, not the target, once you have some history, and note the no-refund caveat: it is a number to plan with, not insurance.
- Your own client and network path, including DNS, egress and timeouts.
- Post-first-token failures, from the previous section.
- Anything that refuses before a provider is called: spend caps, key controls, allowlists, guardrails. INFRO's error reference (opens in a new tab) lists a
gateway_unavailable503 among the codes that surface only when no connection can serve the request.5
In the worked example, a 99.9% gateway, the 99.947% chain and a 99.9% post-first-token term multiply to about 99.75%. The gateway term is larger than the whole chain's failure budget. Adding a third provider does almost nothing to that number; hardening the layers in front of the chain does far more. Whether the application should fail open to a direct provider call when the gateway is down is a separate design choice, covered in gateway down: fail open or fail closed.
Correlated failures: where β comes from
The common-mode term is not abstract. These are the places failures line up:
- A shared cloud or region. The same model family may be offered by its maker and by a hyperscaler, and two hyperscalers may both be fronted by the same CDN, identity service or DNS. SLAs are scoped per region or per project,12 so a regional event is one outage in two contracts. Check where each of your connections is actually hosted before calling them independent.
- Same-model failover. Routing the same model through two hosts removes host failures but keeps model-level faults: a bad snapshot, a safety-filter regression or a capacity shortage on that model.
- Shared quota or account. Two connections that are the same organization's keys share its rate limit. INFRO's connection failover spreads requests across a provider's active keys and tries standbys last,3 but if the limit is per organization, more keys add no capacity.
- Shared demand spikes. When a popular model launches or a competitor has an incident, traffic moves and several providers throttle at once.
- Your own dependencies. The same egress NAT, secret store or deploy pipeline sits under every target.
A cross-model fallback has a different weakness. It decorrelates model-level faults, but it changes answers, so "available" now depends on whether the fallback is good enough for that feature; test it with fallback equivalence before counting its successes as good events.
Measure the inputs from your own records
You need per-attempt outcomes, not just final outcomes: which target was tried, in what order, and how it failed. INFRO's request records name the model, the connection that served each request and the route,6 and GET /v1/requests and a CSV export of up to 50,000 rows per file are available, bounded by your plan's queryable history of 7 to 365 days.7 OpenTelemetry spans can also be pushed to your own collector, but delivery is best effort with one retry and dropped spans on persistent failure, so the documentation says not to use that channel as the sole accounting record.7 For an SLO, export the request records on a schedule and keep them yourself, because a 7-day window will not cover a 90-day estimate.
A minimal estimator over per-minute data (column names are illustrative; map them from your export):
import pandas as pd
# minute, provider, attempts, failures (aggregated from your exported request records)
df = pd.read_csv("attempts_by_minute.csv", parse_dates=["minute"])
df["down"] = (df.failures / df.attempts) > 0.05 # Vertex-style 5% minute rule, as a yardstick
wide = df.pivot(index="minute", columns="provider", values="down").fillna(False)
a_down, b_down = wide["provider_a"], wide["provider_b"]
both = (a_down & b_down).sum()
rarer = min(a_down.sum(), b_down.sum())
print("minutes both down:", both, "of", rarer, "-> beta ~", both / rarer if rarer else None)Report the counts with the ratio. A β of 0.1 from 3 incident minutes is a guess; from 300 it is a measurement. Re-estimate it quarterly and after any change in where targets are hosted.
From composite to commitment and error budget
Do not commit to the computed number. Subtract a margin for what the model cannot see: unmeasured correlation, a future provider with a worse year, and the variance of a short window. A common practice is to commit to a target whose error budget is at least about twice what the model predicts you will spend. In the example, the model predicts about 99.75%, so a 99.5% commitment leaves roughly half of its budget unspent.
Then state the budget in units people can act on. Over a 30-day month (43,200 minutes):
| Target | Budget (fraction) | Budget in minutes of full outage |
|---|---|---|
| 99.5% | 0.5% | 216 |
| 99.9% | 0.1% | 43.2 |
| 99.95% | 0.05% | 21.6 |
Because AI traffic is bursty and uneven, express the budget as a fraction of requests, not minutes, so a quiet night does not hide a bad hour. Alert on burn rate: page when a short window and a longer window are both consuming budget much faster than the month allows, and open a ticket on a slow burn. When the budget is spent, the policy is the usual one: freeze risky changes to routes, models and prompts until it recovers.
Finally, write the assumptions on the SLO page itself: the SLI definition, the window used to estimate each term, the β and how it was obtained, and which refusals are excluded. When any of those change, the number changes.
Building the chain and the record in INFRO
INFRO can supply the mechanics this method needs: routes (opens in a new tab) with priority or weighted targets over your own connections, which still fall through to the rest when the first pick fails,6 failover and fallbacks (opens in a new tab) before the first token, and per-request records with the connection that served each attempt for the estimates above. It does not make the chain's failures independent, and its own availability is one of the series terms to include.
