Skip to content

Cost and FinOpsAnalysis

Do providers bill cancelled or failed streams? What to forecast

The streaming pages from OpenAI and Anthropic describe how streams end and fail, but the ones read for this piece don't state what an aborted or errored stream costs. This analysis separates what is documented from what you must measure, and builds a retry-waste line with a low and a high case.

By INFRO Editorial

Published

8 min read

Glowing token streams being compressed into a precise efficient AI infrastructure pipeline

In short

In the pages read for this analysis, neither OpenAI nor Anthropic states whether a cancelled, timed-out or errored stream is billed, and Google's billing page could not be read. Forecast retry waste as a range between "failed attempts cost nothing" and "failed attempts bill in full", then narrow it with your own provider cost reports.

Key takeaways

  • The streaming and error pages from OpenAI and Anthropic read for this piece describe stream mechanics, not billing of interrupted calls, so any billed fraction you put in a forecast is an assumption until your own cost reports confirm it.
  • A dropped connection and a cancelled request are different events: OpenAI's background mode documents that a response keeps running after the client drops, and cancel is a separate call.
  • Retry waste scales with prompt size times the rate of failed attempts, so long-context routes dominate the line even when failure rates are low.
  • Forecast two cases, failed attempts free and failed attempts billed in full, then compare the high case with provider cost reports to see which one your traffic behaves like.
  • Record every attempt with its own request id, not only the final success, or the waste can't be separated from normal spend.

What the documentation actually says

The question is simple: if a stream is aborted, times out or errors halfway, what does the provider bill? The pages read for this analysis don't answer it. OpenAI's streaming guide covers how to enable streaming and which events to listen for: response.created, response.output_text.delta, response.completed and error.1 It says nothing about billing for a stream that never reaches response.completed. Anthropic's streaming page describes the event flow and notes that the token counts in the usage field of message_delta are cumulative.2 Its errors page describes what happens when a call fails, but it doesn't state a charge either.3

Google is a gap in this review. Its Gemini billing page could not be read in this session, so there is no Google row below with a claim in it. If you run Gemini, read that page yourself and fill the row in before using the numbers in this article.

That leaves a forecaster with a choice: guess a billed fraction, or model it as a range and measure. The rest of this piece takes the second route. What the pages do document is useful for deciding which events can produce spend that never reaches a user.

EventOpenAIAnthropicGoogle
Error before the stream startsNot stated in the pages readHTTP error codes, including 500 timeout_error when the server time limit is hit before the response starts3Not read
Error after the stream startserror event in the stream1error event after a 200, such as overloaded_error2Not read
Client disconnectsBackground streams keep running; you can resume from a cursor4Not stated in the pages readNot read
Explicit cancelCancel endpoint for background responses, idempotent4Not stated in the pages readNot read
Billing rule for any of the aboveNot stated in the pages readNot stated in the pages readNot read
What each provider's pages state about interrupted calls. "Not stated" refers only to the pages read on 2026-11-05.

Four events that can bill without delivering

Absent a stated rule, classify events by what the provider's system has probably done by the time your client acts. The documents support some of this and leave the rest to inference, and the inference is labelled below.

  1. Failure before the first token. Anthropic's errors page lists 500, 504 and 529 as possible outcomes, and says the official SDK retries transient failures twice by default, honoring retry-after.3 Those retries are attempts you did not write. If any attempt were billable, the SDK default could multiply your cost without a line in your code.
  2. Error after a 200. Both providers can fail inside an established stream. Anthropic says errors can occur after the API returns a 200, and that standard error handling does not apply then.3 Your retry logic has to detect the in-stream error event, not just the HTTP status. The partial output was generated; whether it is billed is the open question.
  3. Client timeout while generation continues. OpenAI's background mode documents that if the connection drops, "the response continues running" and you can reconnect.4 That is the documented behaviour for background responses. For ordinary foreground streams the pages don't say, so treat this as a risk to test, not a fact.
  4. Cancel that arrives late. OpenAI's cancel call is a separate request, and cancelling twice returns the final response object.4 Between the user leaving and the cancel landing, tokens are being produced. The gap length is your own latency, which you can measure.

A fifth case belongs to anyone running a gateway. INFRO's failover moves a request to the next connection only before the first token, and its docs say every upstream failure before the first token moves on, including 5xx, timeouts and 429s.5 After a mid-stream error event the connection closes without the [DONE] sentinel.6 So a single user request can create several upstream attempts, each with its own possible charge on your own provider account.

The retry-waste line

Put waste in the forecast as its own line with an explicit billed-fraction parameter per failure class. For each class c:

Text
waste_c = requests × failure_rate_c × billed_fraction_c × cost_of_one_failed_attempt_c

Where cost_of_one_failed_attempt is the prompt tokens times the input rate plus the output tokens produced before the failure times the output rate. The billed fraction has three useful values: 0 (the provider bills nothing for a failed attempt), 1 (it bills what was generated up to the failure) and a measured value in between. Run the forecast at 0 and at 1 and report both.

Two inputs deserve care. First, the prompt term repeats on every retry, so long prompts make the line grow with context size even when failure rates are flat. Second, output before failure depends on where the failure lands. Mid-stream failures after the model has produced hidden reasoning (Anthropic's page documents thinking_delta events streaming before text2) can carry more output than the visible text suggests.

The example is deliberately modest. The point is the shape: the line is small on short prompts and large on long-context routes, and the base forecast can't show that because average cost per request blends both.

Measure the billed fraction instead of assuming it

The low and high cases are a bracket. To tighten it, you need attempt-level records and a provider cost report to compare against.

  • Log every attempt. Give each upstream attempt its own record with the provider's request id, the attempt number, how it ended (success, error before first token, error mid-stream, client abort, timeout) and the tokens seen so far. Anthropic's responses include a request-id header for this purpose.3
  • Capture usage on streams that finish. With stream_options.include_usage, INFRO sends a final chunk carrying prompt_tokens, completion_tokens and an estimated cost before [DONE].6 That chunk never arrives on a stream that dies, which is why a stream's absence of a usage chunk is itself a data point: count those.
  • Run a controlled probe. Send a known set of requests, abort a known fraction at a fixed token count, wait for the provider's usage report to settle, and compare totals with and without the aborts. Do it per provider and per model, because one provider's behaviour tells you nothing about another's.
  • Reconcile daily. The gap between your estimate and the provider's reported cost, tracked over weeks, shows whether failed attempts are billed. A gap that rises with your failure rate points at the high case.
attempt_ledger.py
from dataclasses import dataclass

@dataclass
class Attempt:
    request_id: str       # provider's id, not yours
    attempt: int
    outcome: str          # ok | pre_token_error | mid_stream_error | client_abort | timeout
    prompt_tokens: int
    output_tokens_seen: int

def waste_range(attempts, in_rate, out_rate):
    """Return (low, high) spend on attempts that did not succeed.
    Rates are per token. Low assumes failed attempts bill nothing."""
    high = sum(
        a.prompt_tokens * in_rate + a.output_tokens_seen * out_rate
        for a in attempts if a.outcome != "ok"
    )
    return 0.0, high

If you run requests through a gateway, it can supply some of this. INFRO records per-request model, latency, token counts and estimated provider cost, exports them through GET /v1/requests, and compares estimated spend daily against the cost OpenAI's and Anthropic's own cost APIs report. A persistent gap is the signal that something is billed that the estimate did not count. The estimate is not an invoice, so keep the provider's report as the source of truth.

Reduce the line before you measure it

Some controls are free of any billing assumption.

  • Cap SDK retries deliberately. Anthropic's SDK retries twice by default and exposes max_retries.3 Decide that number; don't inherit it. Stacking an SDK retry under a gateway retry under an application retry multiplies attempts.
  • Retry once, at one layer. Put the retry where you can see it, and make the other layers pass failures through.
  • Never resubmit on a timeout without checking. If the provider may still be generating, resubmitting creates a second full attempt. For long jobs, OpenAI documents background mode with polling and resume.4
  • Send cancel promptly. A cancel issued when the user leaves bounds the output the provider produces after that point, where the API supports it.
  • Match timeouts to the first-token time. A timeout shorter than your slow-but-healthy tail turns working requests into retries; see measuring time to first token.

For the surrounding costs of failover itself, such as price gaps between models and cold caches, see what failover does to your bill. For hedging and retry budgets, see should you hedge LLM requests.

What to put in the forecast review

A short checklist for the next forecast cycle:

  • A retry-waste line exists, separate from base spend, with a low and a high case.
  • Each failure class has a rate you measured from attempt logs, not a guess.
  • The billed fraction for each provider is marked "assumed" or "measured on (date)".
  • Long-context routes are forecast separately, because their prompt term dominates.
  • Retry layers are inventoried: SDK, gateway, application, queue worker.
  • Google's billing page, or any provider page not read here, has been checked by someone before its row is filled in.

Provider docs change, and a rule that's unstated today may be stated tomorrow. Re-read the pages when you re-run the probe, and date the result.

Questions readers ask

Is a request I cancel from the client billed?
The pages read for this piece don't say. OpenAI documents a cancel call for background responses and says cancelling twice is idempotent4, but neither it nor the streaming page states a billing rule. Treat the billed fraction as unknown and test it against your provider's cost report.
Should retry waste be part of the base forecast or a separate line?
A separate line. Folded into the base, it hides behind average cost per request. As its own line it has an owner, a failure-rate input and a low and high case that you can tighten as invoices arrive.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Streaming API responses (opens in a new tab) — OpenAI. Accessed Nov 5, 2026.
  2. 2.Streaming messages (opens in a new tab) — Anthropic. Accessed Nov 5, 2026.
  3. 3.Claude API errors (opens in a new tab) — Anthropic. Accessed Nov 5, 2026.
  4. 4.Background mode (opens in a new tab) — OpenAI. Accessed Nov 5, 2026.
  5. 5.Failover & fallbacks (opens in a new tab) — INFRO. Accessed Nov 5, 2026.
  6. 6.Streaming — API reference (opens in a new tab) — INFRO. Accessed Nov 5, 2026.

About this article

Read on 2026-11-05: OpenAI's streaming and background-mode guides, Anthropic's streaming and errors pages, and INFRO's streaming, failover and docs pages. Google's Gemini billing page could not be read in this session, so nothing is claimed about Gemini. No billing rule for interrupted streams is stated in the pages read, so none is asserted.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.