What the documentation actually says
The question is simple: if a stream is aborted, times out or errors halfway, what does the provider bill? The pages read for this analysis don't answer it. OpenAI's streaming guide covers how to enable streaming and which events to listen for: response.created, response.output_text.delta, response.completed and error.1 It says nothing about billing for a stream that never reaches response.completed. Anthropic's streaming page describes the event flow and notes that the token counts in the usage field of message_delta are cumulative.2 Its errors page describes what happens when a call fails, but it doesn't state a charge either.3
Google is a gap in this review. Its Gemini billing page could not be read in this session, so there is no Google row below with a claim in it. If you run Gemini, read that page yourself and fill the row in before using the numbers in this article.
That leaves a forecaster with a choice: guess a billed fraction, or model it as a range and measure. The rest of this piece takes the second route. What the pages do document is useful for deciding which events can produce spend that never reaches a user.
| Event | OpenAI | Anthropic | |
|---|---|---|---|
| Error before the stream starts | Not stated in the pages read | HTTP error codes, including 500 timeout_error when the server time limit is hit before the response starts3 | Not read |
| Error after the stream starts | error event in the stream1 | error event after a 200, such as overloaded_error2 | Not read |
| Client disconnects | Background streams keep running; you can resume from a cursor4 | Not stated in the pages read | Not read |
| Explicit cancel | Cancel endpoint for background responses, idempotent4 | Not stated in the pages read | Not read |
| Billing rule for any of the above | Not stated in the pages read | Not stated in the pages read | Not read |
Four events that can bill without delivering
Absent a stated rule, classify events by what the provider's system has probably done by the time your client acts. The documents support some of this and leave the rest to inference, and the inference is labelled below.
- Failure before the first token. Anthropic's errors page lists 500, 504 and 529 as possible outcomes, and says the official SDK retries transient failures twice by default, honoring
retry-after.3 Those retries are attempts you did not write. If any attempt were billable, the SDK default could multiply your cost without a line in your code. - Error after a 200. Both providers can fail inside an established stream. Anthropic says errors can occur after the API returns a 200, and that standard error handling does not apply then.3 Your retry logic has to detect the in-stream
errorevent, not just the HTTP status. The partial output was generated; whether it is billed is the open question. - Client timeout while generation continues. OpenAI's background mode documents that if the connection drops, "the response continues running" and you can reconnect.4 That is the documented behaviour for background responses. For ordinary foreground streams the pages don't say, so treat this as a risk to test, not a fact.
- Cancel that arrives late. OpenAI's cancel call is a separate request, and cancelling twice returns the final response object.4 Between the user leaving and the cancel landing, tokens are being produced. The gap length is your own latency, which you can measure.
A fifth case belongs to anyone running a gateway. INFRO's failover moves a request to the next connection only before the first token, and its docs say every upstream failure before the first token moves on, including 5xx, timeouts and 429s.5 After a mid-stream error event the connection closes without the [DONE] sentinel.6 So a single user request can create several upstream attempts, each with its own possible charge on your own provider account.
The retry-waste line
Put waste in the forecast as its own line with an explicit billed-fraction parameter per failure class. For each class c:
waste_c = requests × failure_rate_c × billed_fraction_c × cost_of_one_failed_attempt_cWhere cost_of_one_failed_attempt is the prompt tokens times the input rate plus the output tokens produced before the failure times the output rate. The billed fraction has three useful values: 0 (the provider bills nothing for a failed attempt), 1 (it bills what was generated up to the failure) and a measured value in between. Run the forecast at 0 and at 1 and report both.
Two inputs deserve care. First, the prompt term repeats on every retry, so long prompts make the line grow with context size even when failure rates are flat. Second, output before failure depends on where the failure lands. Mid-stream failures after the model has produced hidden reasoning (Anthropic's page documents thinking_delta events streaming before text2) can carry more output than the visible text suggests.
The example is deliberately modest. The point is the shape: the line is small on short prompts and large on long-context routes, and the base forecast can't show that because average cost per request blends both.
Measure the billed fraction instead of assuming it
The low and high cases are a bracket. To tighten it, you need attempt-level records and a provider cost report to compare against.
- Log every attempt. Give each upstream attempt its own record with the provider's request id, the attempt number, how it ended (success, error before first token, error mid-stream, client abort, timeout) and the tokens seen so far. Anthropic's responses include a
request-idheader for this purpose.3 - Capture usage on streams that finish. With
stream_options.include_usage, INFRO sends a final chunk carryingprompt_tokens,completion_tokensand an estimatedcostbefore[DONE].6 That chunk never arrives on a stream that dies, which is why a stream's absence of a usage chunk is itself a data point: count those. - Run a controlled probe. Send a known set of requests, abort a known fraction at a fixed token count, wait for the provider's usage report to settle, and compare totals with and without the aborts. Do it per provider and per model, because one provider's behaviour tells you nothing about another's.
- Reconcile daily. The gap between your estimate and the provider's reported cost, tracked over weeks, shows whether failed attempts are billed. A gap that rises with your failure rate points at the high case.
from dataclasses import dataclass
@dataclass
class Attempt:
request_id: str # provider's id, not yours
attempt: int
outcome: str # ok | pre_token_error | mid_stream_error | client_abort | timeout
prompt_tokens: int
output_tokens_seen: int
def waste_range(attempts, in_rate, out_rate):
"""Return (low, high) spend on attempts that did not succeed.
Rates are per token. Low assumes failed attempts bill nothing."""
high = sum(
a.prompt_tokens * in_rate + a.output_tokens_seen * out_rate
for a in attempts if a.outcome != "ok"
)
return 0.0, highIf you run requests through a gateway, it can supply some of this. INFRO records per-request model, latency, token counts and estimated provider cost, exports them through GET /v1/requests, and compares estimated spend daily against the cost OpenAI's and Anthropic's own cost APIs report. A persistent gap is the signal that something is billed that the estimate did not count. The estimate is not an invoice, so keep the provider's report as the source of truth.
Reduce the line before you measure it
Some controls are free of any billing assumption.
- Cap SDK retries deliberately. Anthropic's SDK retries twice by default and exposes
max_retries.3 Decide that number; don't inherit it. Stacking an SDK retry under a gateway retry under an application retry multiplies attempts. - Retry once, at one layer. Put the retry where you can see it, and make the other layers pass failures through.
- Never resubmit on a timeout without checking. If the provider may still be generating, resubmitting creates a second full attempt. For long jobs, OpenAI documents background mode with polling and resume.4
- Send cancel promptly. A cancel issued when the user leaves bounds the output the provider produces after that point, where the API supports it.
- Match timeouts to the first-token time. A timeout shorter than your slow-but-healthy tail turns working requests into retries; see measuring time to first token.
For the surrounding costs of failover itself, such as price gaps between models and cold caches, see what failover does to your bill. For hedging and retry budgets, see should you hedge LLM requests.
What to put in the forecast review
A short checklist for the next forecast cycle:
- A retry-waste line exists, separate from base spend, with a low and a high case.
- Each failure class has a rate you measured from attempt logs, not a guess.
- The billed fraction for each provider is marked "assumed" or "measured on (date)".
- Long-context routes are forecast separately, because their prompt term dominates.
- Retry layers are inventoried: SDK, gateway, application, queue worker.
- Google's billing page, or any provider page not read here, has been checked by someone before its row is filled in.
Provider docs change, and a rule that's unstated today may be stated tomorrow. Re-read the pages when you re-run the probe, and date the result.
