Streaming LLM failures: interrupted streams and recovery

With a non-streaming request, the status code is the verdict: 200 means you have the whole answer. With streaming, the server has to send its status line before the model has produced anything, so the 200 only says the stream opened. Everything that goes wrong afterwards — the provider overloads, a proxy drops an idle connection, the model goes quiet — has to be detected from the body.

Anthropic states the consequence directly: when a streamed response is in progress, an error can arrive after the API has already returned 200, and it does not follow the normal error-handling path¹. Most client code that "handles errors" only handles the status code, and so records interrupted answers as successes.

Why a stream fails differently

Streaming LLM APIs use server-sent events: one long HTTP response carrying a sequence of data: lines, each event terminated by a blank line². Three properties of the format shape every failure:

• The status is already sent. Any error after the first byte has to be expressed as an event, or by closing the connection.
• An incomplete event is discarded. If the connection ends in the middle of an event, before its blank line, the specification says the partial event is not dispatched². A torn last line is lost, not half-parsed.
• Silence is legal. Nothing in the format bounds the gap between events, and the specification notes that some legacy proxies drop connections that stay idle, suggesting a comment line every 15 seconds or so to keep them open². A model that thinks for a minute before its next token looks, on the wire, exactly like a dead connection.

The four ways a stream ends

Every stream ends in one of four states, and only one of them is a complete answer.

Each API marks a clean finish explicitly, which is what makes detection possible:

Anthropic also asks clients to handle unknown event types gracefully, because new ones may be added³. A parser that throws on an event it has not seen turns a provider's additive change into your outage. Skip what you do not recognize; fail only on what you do.

The failures you will actually meet

• A provider error mid-generation. The provider is overloaded or fails after it started. On Anthropic's API this arrives as an error event with the same types as HTTP errors³.
• A dropped connection. A proxy, load balancer, mobile network or process restart closes the socket. There is no error event: the stream simply ends without its terminal marker.
• A stall. The connection stays open and nothing arrives. Without an idle timeout the request, its socket and whatever the user is looking at wait indefinitely.
• A truncated answer. The stream finishes cleanly with finish_reason: "length". The transport worked; the content is incomplete. Treat it as a distinct outcome, not a transient error.
• A policy stop. A content filter or guardrail ends the response. Retrying the same request produces the same stop.
• Partial tool arguments. Tool-call arguments stream as fragments of JSON and are only valid once the call finishes³. A stream that dies mid-call leaves a string that may parse, and still be wrong.

There is also a quieter cost. OpenAI points out that streaming makes output harder to moderate, because partial completions are harder to evaluate and moderation scores arrive only after the full output exists⁴. Whatever you showed before the failure was shown.

Detect the ending in the client

The reader below consumes an OpenAI-compatible chat stream and returns one of four outcomes rather than a string. Its idle timeout is httpx's read timeout, which bounds the wait for each read rather than the whole response, so a long answer that keeps arriving is never cut off while a silent one is.

import json
from dataclasses import dataclass
from typing import Callable

import httpx


@dataclass
class StreamResult:
    outcome: str                      # "complete" | "error" | "closed" | "stalled"
    text: str
    finish_reason: str | None = None
    error: dict | None = None
    usage: dict | None = None


def read_chat_stream(client: httpx.Client, url: str, api_key: str, body: dict, *,
                     idle_s: float = 60.0,
                     on_text: Callable[[str], None] | None = None) -> StreamResult:
    parts: list[str] = []
    finish: str | None = None
    usage: dict | None = None
    saw_done = False
    timeout = httpx.Timeout(10.0, read=idle_s)  # read= is the idle limit between chunks

    def result(outcome: str, error: dict | None = None) -> StreamResult:
        return StreamResult(outcome, "".join(parts), finish, error, usage)

    try:
        with client.stream("POST", url, json={**body, "stream": True},
                           headers={"Authorization": f"Bearer {api_key}"},
                           timeout=timeout) as resp:
            if resp.status_code >= 400:
                resp.read()
                resp.raise_for_status()   # failed before any event: an ordinary HTTP error
            for line in resp.iter_lines():
                if not line.startswith("data:"):
                    continue              # blank separators, ": comments", "event:" lines
                data = line[5:].strip()
                if data == "[DONE]":
                    saw_done = True
                    continue
                try:
                    event = json.loads(data)
                except json.JSONDecodeError:
                    return result("closed")   # a torn final line: the stream was cut
                if "error" in event:
                    return result("error", event["error"])
                if event.get("usage"):
                    usage = event["usage"]
                choices = event.get("choices") or []
                if not choices:
                    continue              # the usage chunk has no choices
                delta = (choices[0].get("delta") or {}).get("content")
                if delta:
                    parts.append(delta)
                    if on_text:
                        on_text(delta)
                if choices[0].get("finish_reason"):
                    finish = choices[0]["finish_reason"]
    except httpx.ReadTimeout:
        return result("stalled")
    except (httpx.RemoteProtocolError, httpx.ReadError):
        return result("closed")

    if saw_done and finish is not None:
        return result("complete")
    return result("closed")               # end of file without the terminator

The rule the function encodes is the whole point: complete requires both a finish reason and the terminator. Any other ending returns whatever text arrived, labelled as what it is, so the caller cannot mistake it for an answer.

Decide what to do with the partial output

What happens next depends on what the output was for.

A retry is a new generation. It may produce different text, a different length and different tool calls, so the retry must replace the partial output rather than append to it. Appending the second attempt to the first produces an answer no model wrote.

import random
import time

NO_RETRY_TYPES = {"invalid_request_error", "authentication_error", "budget_exceeded",
                  "model_not_found", "permission_denied"}
NO_RETRY_CODES = {"guardrail_blocked", "guardrail_unavailable"}


def worth_retrying(r: StreamResult) -> bool:
    if r.outcome == "complete":
        return False
    if r.outcome == "error" and r.error:
        return (r.error.get("type") not in NO_RETRY_TYPES
                and r.error.get("code") not in NO_RETRY_CODES)
    return True                           # closed or stalled


def stream_with_retry(client, url, api_key, body, render, *, attempts: int = 3) -> StreamResult:
    """render: your UI or buffer, with reset(), write(text) and mark_interrupted(result)."""
    for attempt in range(attempts):
        render.reset()                    # replace, never append
        r = read_chat_stream(client, url, api_key, body, on_text=render.write)
        if not worth_retrying(r) or attempt == attempts - 1:
            if r.outcome != "complete":
                render.mark_interrupted(r)
            return r
        time.sleep(random.uniform(0, min(8.0, 0.5 * 2 ** attempt)))  # full jitter
    raise RuntimeError("unreachable")

For a chat interface, a visible "interrupted, regenerating" state is better than text that silently rewrites itself under the reader. For a background job, discard and regenerate. For anything that calls a tool or writes data, never act on output from a stream that did not complete.

Anthropic documents one alternative to regenerating from scratch: capture the partial text and send a continuation request, as an assistant prefix for Claude 4.5 and earlier models or as a user instruction to continue for 4.6 and later³. It saves regenerating what already arrived, at the price of a second billed request whose input includes the partial answer, and it does not work for tool-use or thinking blocks, which cannot be partially recovered³. Use it for long text where a seam is acceptable; regenerate everywhere else.

What INFRO sends when a stream fails

INFRO streams chat completions in the OpenAI-compatible format, so the reader above works against it unchanged (streaming). Its behaviour splits at the first event:

• Before the first event, a provider failure does not reach you. The gateway reads the upstream's first event inside the attempt, so a provider that returns an error, or fails or falls silent before it has produced one, is abandoned and the request moves to the next of your connections, or the next target on the route. The only symptom is a later first chunk. If nothing can serve, you get a normal HTTP error status with a JSON body (errors).
• After the first event, INFRO does not restart the response on another provider. If the serving provider drops, it sends a final event whose payload is the standard error object and closes the stream without data: [DONE]; the usage chunk you asked for never arrives.
• A stall longer than the connection's timeout, and never less than 120 seconds, ends the stream the same way, with type request_timeout and code stream_stalled. The floor is deliberate: reasoning models can pause for a long time between events. If your interface needs a shorter limit, set it in the client, as the reader above does.
• An output guardrail holds back the last few characters before releasing them, and a block match ends the stream with guardrail_blocked. Text sent before the match was already delivered, and a retry will be stopped the same way.

Every chunk carries the request id, also returned in the X-INFRO-Request-Id header. GET /v1/requests/{id} then shows the connection that served, the attempts that failed before the stream opened, and the error that ended it, so an interrupted answer in a support ticket can be traced to the provider account that dropped it (traces).

Checklist

• Treat a stream as complete only after its terminal event: [DONE], message_stop or response.completed.
• Set an idle timeout between events as well as a connect timeout, sized for the slowest model you call.
• Skip unknown event types and comment lines instead of failing on them.
• Return an outcome, not a string, from your stream reader, and record the outcome.
• Discard partial tool-call arguments; never execute a call from an incomplete stream.
• Replace partial output on retry, and show an interrupted state in any interface that rendered it.
• Exclude guardrail, authentication, budget and invalid-request errors from automatic retry.
• Handle finish_reason: "length" as a truncated answer, not a transient failure.
• Log the request id of every interrupted stream so it can be traced later.