What does failover do to your bill? Price gaps and cold caches

Where a failover touches the invoice

A failover is a routing decision, but your provider bills what it executed. Four things change when traffic moves from the primary to a fallback, and each one lands on a different line of the invoice:

1. The rate. The fallback model has its own input, output and cache rates.
2. The cache. Prompt caches live with the upstream that served the request, so a failover resets the cache for that prefix and a cold prefix is billed as a cold prefix.¹ A different model is a different cache again, so the prefix is written in full on the new one.
3. The attempts. Requests that time out or fail before the response starts are retried elsewhere, so one user request can become several upstream attempts.²
4. The way back. When the primary recovers, its cache has usually expired, so you pay a second cold start in the opposite direction.

The existing posts on failover drills, circuit breakers and the September 3 incidents cover when and how to switch. This one covers what the switch costs, so a FinOps lead can put a number on an incident before it happens. That fits the FinOps Framework's Planning & Estimating and Forecasting capabilities, which sit under the Quantify Business Value domain.³

How INFRO's failover decides what gets retried

The mechanics matter because they set the multiplier on attempts. Connection failover is same-model: with a model named directly, INFRO tries every enabled, healthy connection that serves that model's vendor, and within one provider it spreads requests across active keys. Model fallbacks are cross-model and opt-in through a fallbacks array of up to five catalog model IDs, tried once every connection for the primary has failed before the first token.²

Two details change the cost picture:

• Every upstream failure before the first token moves to the next connection: a 5xx, a timeout, a 429, a rejected credential, a 404, even a 400 or 422.² A provider 429 does not count against the connection's health, so a rate-limited primary keeps being tried first on later requests.
• INFRO's own refusals, such as a 402 budget error or a 403 allowlist error, are decided before any provider is called and return immediately without failover.²

Failover also stops at the first token. Once a stream has started, a mid-stream error closes the connection without the [DONE] sentinel and your client sees it.⁴ If your application retries that request, the retry pays for the input again, and the partial output of the first attempt was already generated. Mid-stream failures are the expensive failure mode, because the gateway cannot hide them and your own retry logic takes over.

The price gap: the rate ratio times the traffic

This is usually the biggest term. Take the Anthropic list rates, which are what the customer's own provider bills.⁵ A support assistant has a 20,000-token cacheable prefix, 1,000 tokens of fresh input and 600 output tokens per request.

Two readings of the table. First, the ratio between warm costs is the price gap, and it applies to every request for as long as the fallback serves traffic. Second, on every model a cold request costs roughly five to six times a warm one, because the prefix is written at a premium instead of read at a fraction of input.

Check the ratio before you list a model in fallbacks. Haiku 5.5 also has a second tier for prompts over 100,000 tokens ($0.50 input, $2.50 output),⁵ so a long-context request that falls back to a small model can cost more per token than the same model on short prompts.

The cold cache, twice

A cache write on Anthropic's models costs 1.25 times the input rate for a five-minute entry, and a read costs 0.1 times, or 0.05 times on Opus 5.5 and Sonnet 5.5.⁶ OpenAI reads are roughly 0.1 times input, and on GPT-5.6 and later writes are 1.25 times input.¹ Providers change these in both directions, so recompute from your own rates.

What this means in an incident:

• Onset. Every live prefix on the fallback is cold. If you have many long-lived conversations or per-tenant prefixes, each one pays a write on its next turn. A hot shared prefix pays a handful of writes, since the prefix is cached once the response starts and requests racing ahead of that first response also miss.⁶
• Recovery. The primary's entries lapse after about five minutes without a read, so each prefix pays a write on return. Reads refresh the entry at no additional charge, so a prefix that keeps being read stays warm.⁶
• Flapping. If health checks bounce traffic between the two, each switch can repeat the write for prefixes whose entry on the other side lapsed. Switches shorter than the cache lifetime are cheaper.
• No assumed relief on an outage. Providers may offer ways to avoid paying the cache write twice when a refused request is retried on another model. Do not assume such a mechanism covers an outage failover through a gateway. Budget for the write.

Retries and timeouts: the multiplier you can control

Attempts multiply across layers. If your application retries up to 3 times, each attempt can try 2 connections for the primary and then 2 for a fallback model, so one user request can generate up to 12 upstream attempts, and more if the fallbacks list is longer (the limit is five models).² Most of these attempts fail fast and cost nothing. The ones that cost money are slow failures: a request that the provider started working on and your timeout abandoned.

I did not confirm how each provider bills requests abandoned by the client, so test it. Send a request with a short client timeout, then compare your provider's usage export with your gateway log for that request. The gateway's per-request record carries model, latency, token counts and estimated provider cost,⁷ which gives you the other side of that comparison.

Worked example: a three-hour incident

Three conclusions follow from the arithmetic, which holds as a shape even if your numbers differ:

• The rate ratio dominates: in the Opus case it is $540 of the $626 extra, and in the Fable case it is $1,890 of the $2,090.
• Cold caches are the second term, here 14% of the extra in the Opus case. It grows with the number of live prefixes and with prefix length, and a single shared prefix makes it nearly vanish.
• Timeouts are the smallest term unless your prompts are long, your outputs are long, or retry layers multiply.

def incident_extra(reqs, warm_fb, cold_fb, warm_pri, cold_pri, live_prefixes):
    """Extra spend vs. no incident. Costs are per request, in dollars."""
    steady = reqs * warm_fb
    onset = live_prefixes * (cold_fb - warm_fb)       # fallback cold start
    recovery = live_prefixes * (cold_pri - warm_pri)  # primary cold start
    return steady + onset + recovery - reqs * warm_pri

print(incident_extra(54_000, 0.020, 0.116, 0.010, 0.058, 600))  # 626.4

Controls to set before the incident

The budget you enforce during the incident matters as much as the estimate. A hard cap per project is checked before the provider call, and a request refused with a 402 returns immediately without failover.²⁸ If a project's budget was sized for the primary's rate, a fallback at 2x or 4.5x can exhaust it mid-incident and turn a degraded service into an outage. Decide deliberately whether the cap or the availability wins, and size it with the fallback price in mind.

• List each fallback with its warm and cold per-request cost next to the primary's, and reject any whose ratio you would not accept for a full day.
• Use model allowlists: a fallback entry is skipped without an error when the allowlist does not permit it,² so the allowlist is a cost control as well as a policy one.
• Enter your negotiated rates so estimates use what your contracts carry, not list rates. Estimates are what you will read during the incident, and your provider's invoice is what you pay.
• Cap application-level retries and set the timeout against your longest normal response, so slow failures do not multiply across layers.
• Count live prefixes. The cold-start term is that number times the write premium, on both sides of the incident.
• Watch for the signature: INFRO's traffic-change findings include more fallbacks and a lost prompt cache, each with an estimated weekly cost change, and they are built from request metadata, not answers. Spend anomaly alerts can route to email, Slack and webhooks.

After the incident, reconcile. Compare the estimated cost per request for the window with your provider's usage and cost report for the same hours, since a persistent gap points at a rate or cache-accounting assumption that was wrong. The FinOps principle that data should be accessible, timely and accurate applies here: during an incident the estimate is the timely number, and the invoice is the accurate one.³ For the break-even of the caching side itself, see prompt caching across providers.