# What does failover do to your bill? Price gaps and cold caches

URL: https://infro.io/blog/what-does-failover-do-to-your-bill-price-gaps-and-cold-caches
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-03
Last modified: 2026-11-03
Last reviewed: 2026-11-03 (next review due 2026-12-03)
Topic: Cost and FinOps
Format: Analysis

> An incident costs the price ratio of your fallback model, times the traffic it carries, plus one cold-cache write per live prefix on the way in and again on the way out. Timeouts and retries are usually the smallest term, unless your prompts are long or your retry layers multiply.

Key takeaways:
- The largest term in an incident is usually the price ratio between the fallback and the primary model, applied to every request for the whole outage.
- Prompt caches are per model and per upstream, so a failover pays a cache write on the fallback and another on the primary when traffic returns.
- A request that fails before the first token is the cheap failure; a stream that dies after it cannot fail over and is paid for twice if your app retries.
- Retry layers multiply: application retries, connections per model and up to five fallback models can turn one user request into many upstream attempts.
- Price your fallback list before the incident and put the estimate next to the budget that will be enforced during it.

## Where a failover touches the invoice

A failover is a routing decision, but your provider bills what it executed. Four things change when traffic moves from the primary to a fallback, and each one lands on a different line of the invoice:

1. **The rate.** The fallback model has its own input, output and cache rates.
2. **The cache.** Prompt caches live with the upstream that served the request, so a failover resets the cache for that prefix and a cold prefix is billed as a cold prefix.[1] A different model is a different cache again, so the prefix is written in full on the new one.
3. **The attempts.** Requests that time out or fail before the response starts are retried elsewhere, so one user request can become several upstream attempts.[2]
4. **The way back.** When the primary recovers, its cache has usually expired, so you pay a second cold start in the opposite direction.

The existing posts on [failover drills](https://infro.io/blog/llm-failover-drills), [circuit breakers](https://infro.io/blog/llm-circuit-breakers-throttling) and the [September 3 incidents](https://infro.io/blog/september-3-2026-ai-outages-failover) cover when and how to switch. This one covers what the switch costs, so a FinOps lead can put a number on an incident before it happens. That fits the FinOps Framework's Planning & Estimating and Forecasting capabilities, which sit under the Quantify Business Value domain.[3]

## How INFRO's failover decides what gets retried

The mechanics matter because they set the multiplier on attempts. Connection failover is same-model: with a model named directly, INFRO tries every enabled, healthy connection that serves that model's vendor, and within one provider it spreads requests across active keys. Model fallbacks are cross-model and opt-in through a `fallbacks` array of up to five catalog model IDs, tried once every connection for the primary has failed before the first token.[2]

Two details change the cost picture:

- Every upstream failure before the first token moves to the next connection: a 5xx, a timeout, a 429, a rejected credential, a 404, even a 400 or 422.[2] A provider 429 does not count against the connection's health, so a rate-limited primary keeps being tried first on later requests.
- INFRO's own refusals, such as a 402 budget error or a 403 allowlist error, are decided before any provider is called and return immediately without failover.[2]

Failover also stops at the first token. Once a stream has started, a mid-stream error closes the connection without the `[DONE]` sentinel and your client sees it.[4] If your application retries that request, the retry pays for the input again, and the partial output of the first attempt was already generated. Mid-stream failures are the expensive failure mode, because the gateway cannot hide them and your own retry logic takes over.

## The price gap: the rate ratio times the traffic

This is usually the biggest term. Take the Anthropic list rates, which are what the customer's own provider bills.[5] A support assistant has a 20,000-token cacheable prefix, 1,000 tokens of fresh input and 600 output tokens per request.

> **Example: Illustrative per-request cost, Anthropic list rates**
> The workload shape is made up for illustration. Rates are per million tokens from Anthropic's pricing page: input and output, then 5-minute cache write and cache read.[5]
> 
> | Model | Input / output | 5m write / read | Warm request | Cold request | Warm vs Sonnet 5.5 |
> | --- | --- | --- | --- | --- | --- |
> | Sonnet 5.5 (primary) | $2 / $10 | $2.50 / $0.10 | $0.0100 | $0.0580 | 1.0x |
> | Opus 5.5 | $4 / $20 | $5 / $0.20 | $0.0200 | $0.1160 | 2.0x |
> | Fable 5.1 | $10 / $50 | $12.50 / $0.25 | $0.0450 | $0.2900 | 4.5x |
> | Haiku 5.5 (prompts up to 100K) | $0.10 / $0.50 | $0.125 / $0.01 | $0.0006 | $0.0029 | 0.06x |
> 
> Table: a warm request reads the 20,000-token prefix from cache; a cold one writes it. Both include 1,000 fresh input and 600 output tokens.

Two readings of the table. First, the ratio between warm costs is the price gap, and it applies to every request for as long as the fallback serves traffic. Second, on every model a cold request costs roughly five to six times a warm one, because the prefix is written at a premium instead of read at a fraction of input.

Check the ratio before you list a model in `fallbacks`. Haiku 5.5 also has a second tier for prompts over 100,000 tokens ($0.50 input, $2.50 output),[5] so a long-context request that falls back to a small model can cost more per token than the same model on short prompts.

## The cold cache, twice

A cache write on Anthropic's models costs 1.25 times the input rate for a five-minute entry, and a read costs 0.1 times, or 0.05 times on Opus 5.5 and Sonnet 5.5.[6] OpenAI reads are roughly 0.1 times input, and on GPT-5.6 and later writes are 1.25 times input.[1] Providers change these in both directions, so recompute from your own rates.

What this means in an incident:

- **Onset.** Every live prefix on the fallback is cold. If you have many long-lived conversations or per-tenant prefixes, each one pays a write on its next turn. A hot shared prefix pays a handful of writes, since the prefix is cached once the response starts and requests racing ahead of that first response also miss.[6]
- **Recovery.** The primary's entries lapse after about five minutes without a read, so each prefix pays a write on return. Reads refresh the entry at no additional charge, so a prefix that keeps being read stays warm.[6]
- **Flapping.** If health checks bounce traffic between the two, each switch can repeat the write for prefixes whose entry on the other side lapsed. Switches shorter than the cache lifetime are cheaper.
- **No assumed relief on an outage.** Providers may offer ways to avoid paying the cache write twice when a refused request is retried on another model. Do not assume such a mechanism covers an outage failover through a gateway. Budget for the write.

## Retries and timeouts: the multiplier you can control

Attempts multiply across layers. If your application retries up to 3 times, each attempt can try 2 connections for the primary and then 2 for a fallback model, so one user request can generate up to 12 upstream attempts, and more if the `fallbacks` list is longer (the limit is five models).[2] Most of these attempts fail fast and cost nothing. The ones that cost money are slow failures: a request that the provider started working on and your timeout abandoned.

I did not confirm how each provider bills requests abandoned by the client, so test it. Send a request with a short client timeout, then compare your provider's usage export with your gateway log for that request. The gateway's per-request record carries model, latency, token counts and estimated provider cost,[7] which gives you the other side of that comparison.

> **Warning: Streaming changes the math**
> A request that fails before the first token is a clean retry. A stream that dies at token 400 has produced billable output, cannot fail over, and its retry pays the full prompt again. If you stream long outputs, size your timeout and retry policy for this case, not the pre-token case.[4]

## Worked example: a three-hour incident

> **Example: Illustrative incident estimate**
> Assumptions, all invented for this example: 600 active conversations, each sending a turn every 2 minutes (300 requests per minute, 18,000 per hour); the per-request shape and rates from the table above; the primary model is unavailable for 3 hours (54,000 requests); a health-check detection window of 4 minutes; every conversation's prefix is cold at onset and again at recovery.
> 
> | Scenario | Steady fallback | Cold start on fallback | Cold start on return | Total | Baseline (no incident) | Extra |
> | --- | --- | --- | --- | --- | --- | --- |
> | Fallback to Opus 5.5 | $1,080.00 | $57.60 | $28.80 | $1,166.40 | $540.00 | $626.40 |
> | Fallback to Fable 5.1 | $2,430.00 | $171.00 | $28.80 | $2,629.80 | $540.00 | $2,089.80 |
> 
> Table: arithmetic. Steady = 54,000 x warm cost. Fallback cold start = 600 x (cold − warm) on the fallback, for example 600 x ($0.116 − $0.020) = $57.60. Return = 600 x ($0.058 − $0.010) = $28.80 on Sonnet 5.5.
> 
> The detection window adds little. Suppose 1,200 requests arrive in the 4 minutes and 360 of them time out after 600 output tokens were generated, and suppose the provider bills that work. At $0.010 each that is $3.60.

Three conclusions follow from the arithmetic, which holds as a shape even if your numbers differ:

- The rate ratio dominates: in the Opus case it is $540 of the $626 extra, and in the Fable case it is $1,890 of the $2,090.
- Cold caches are the second term, here 14% of the extra in the Opus case. It grows with the number of live prefixes and with prefix length, and a single shared prefix makes it nearly vanish.
- Timeouts are the smallest term unless your prompts are long, your outputs are long, or retry layers multiply.

```python
def incident_extra(reqs, warm_fb, cold_fb, warm_pri, cold_pri, live_prefixes):
    """Extra spend vs. no incident. Costs are per request, in dollars."""
    steady = reqs * warm_fb
    onset = live_prefixes * (cold_fb - warm_fb)       # fallback cold start
    recovery = live_prefixes * (cold_pri - warm_pri)  # primary cold start
    return steady + onset + recovery - reqs * warm_pri

print(incident_extra(54_000, 0.020, 0.116, 0.010, 0.058, 600))  # 626.4
```

## Controls to set before the incident

The budget you enforce during the incident matters as much as the estimate. A hard cap per project is checked before the provider call, and a request refused with a 402 returns immediately without failover.[2][8] If a project's budget was sized for the primary's rate, a fallback at 2x or 4.5x can exhaust it mid-incident and turn a degraded service into an outage. Decide deliberately whether the cap or the availability wins, and size it with the fallback price in mind.

- [ ] List each fallback with its warm and cold per-request cost next to the primary's, and reject any whose ratio you would not accept for a full day.
- [ ] Use model allowlists: a fallback entry is skipped without an error when the allowlist does not permit it,[2] so the allowlist is a cost control as well as a policy one.
- [ ] Enter your negotiated rates so estimates use what your contracts carry, not list rates. Estimates are what you will read during the incident, and your provider's invoice is what you pay.
- [ ] Cap application-level retries and set the timeout against your longest normal response, so slow failures do not multiply across layers.
- [ ] Count live prefixes. The cold-start term is that number times the write premium, on both sides of the incident.
- [ ] Watch for the signature: INFRO's traffic-change findings include more fallbacks and a lost prompt cache, each with an estimated weekly cost change, and they are built from request metadata, not answers. Spend anomaly alerts can route to email, Slack and webhooks.

After the incident, reconcile. Compare the estimated cost per request for the window with your provider's usage and cost report for the same hours, since a persistent gap points at a rate or cache-accounting assumption that was wrong. The FinOps principle that data should be accessible, timely and accurate applies here: during an incident the estimate is the timely number, and the invoice is the accurate one.[3] For the break-even of the caching side itself, see [prompt caching across providers](https://infro.io/blog/prompt-caching-break-even).

## Questions

### Does a failed attempt before the first token cost anything?

That depends on your provider's billing rules for the failure, and I did not confirm them for every provider. A connection error or 5xx before any work starts is typically a non-event, but a timeout may have consumed compute. Check one known timeout against your provider's usage export before assuming either way.

### Should I fall back to a cheaper-than-primary model to limit incident cost?

Only if it passes your quality test. A smaller fallback lowers the per-request price, but it changes answers, so test equivalence first and watch for prompts that cross a price tier, such as Haiku 5.5's higher rate above 100,000 tokens.

## Sources

1. Prompt caching — INFRO Docs. https://infro.io/docs/features/prompt-caching (accessed 2026-11-03)
2. Failover & fallbacks — INFRO Docs. https://infro.io/docs/features/fallbacks (accessed 2026-11-03)
3. FinOps Framework overview — FinOps Foundation. https://www.finops.org/framework/ (accessed 2026-11-03)
4. Streaming — INFRO Docs. https://infro.io/docs/api/streaming (accessed 2026-11-03)
5. Plans & pricing (API rates) — Anthropic. https://claude.com/pricing (accessed 2026-11-03)
6. Prompt caching — Anthropic. https://platform.claude.com/docs/en/build-with-claude/prompt-caching (accessed 2026-11-03)
7. INFRO documentation overview — INFRO Docs. https://infro.io/docs (accessed 2026-11-03)
8. Errors — INFRO Docs. https://infro.io/docs/api/errors (accessed 2026-11-03)