# Streaming

> Stream chat completions from INFRO over SSE: chunk format, usage and cost reporting, incremental tool calls, and mid-stream error handling.

Source: https://infro.io/docs/api/streaming

> **Note:** INFRO is in private early access. Running these examples requires an early-access account, an active trial or subscription entitlement, and your own provider connection. Self-serve purchasing is not generally open. See [Early access](https://infro.io/early-access).

Set `stream: true` on any [chat completions](https://infro.io/docs/api/chat-completions) request and INFRO returns the response as a stream of server-sent events (SSE) instead of a single JSON body. Tokens arrive as they are generated, so you can render output immediately.

The wire format is OpenAI-compatible: `chat.completion.chunk` events with incremental `delta` payloads, terminated by `data: [DONE]`. Any OpenAI SDK consumes it unchanged — point the client at `https://api.infro.io/v1` as shown in [Quickstart](https://infro.io/docs/quickstart). This page covers the raw protocol for anyone parsing SSE by hand, plus INFRO-specific details: the cost estimate in the final usage chunk, incremental tool-call arguments, and mid-stream error behavior.

## Requesting a stream

Two request fields control streaming. Everything else — routing, fallbacks, tools, response formats — works exactly as it does without streaming.

- `stream` (boolean) — Set to `true` to receive the response as an SSE stream of `chat.completion.chunk` events. Defaults to `false`.
- `stream_options` (object) — Streaming-only options. Ignored unless `stream` is `true`.
- `stream_options.include_usage` (boolean) — When `true`, INFRO sends one extra chunk before `data: [DONE]` whose `usage` object carries `prompt_tokens`, `completion_tokens`, and `cost` — an estimate of what your provider will bill. That chunk's `choices` array is empty.

```bash
curl https://api.infro.io/v1/chat/completions \
  -N \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{"role": "user", "content": "Explain SSE in one paragraph."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'
```

The `-N` flag disables curl's output buffering so events print as they arrive. The response has `Content-Type: text/event-stream` and carries the standard `X-RateLimit-*` headers described in [Limits](https://infro.io/docs/api/limits).

## Reading the event stream

Each event is a single line of the form `data: {json}`, separated by blank lines. The first chunk sets `delta.role` to `assistant`; later chunks carry `delta.content` fragments; the final chunk for a choice has an empty delta and a non-null `finish_reason`. A successful stream ends with the literal sentinel `data: [DONE]` — it is not JSON, so match it before parsing. After a mid-stream error event the connection closes without the sentinel, so treat connection close as terminal too (see [Errors](https://infro.io/docs/api/errors)).

```text
data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[{"index":0,"delta":{"content":"Server"},"finish_reason":null}]}

data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[{"index":0,"delta":{"content":"-sent events are..."},"finish_reason":null}]}

data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481601,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
```

A single chunk, formatted for readability:

```json
{
  "id": "req_2mV9kQ7xT4bN8wRz",
  "object": "chat.completion.chunk",
  "created": 1787481600,
  "model": "anthropic/claude-sonnet-5",
  "route": "primary",
  "route_name": null,
  "provider": "anthropic",
  "connection": "conn_8Kx2mQ4vR7tYb1Zp",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "-sent events are..."
      },
      "finish_reason": null
    }
  ]
}
```

- `id` (string) — The INFRO request ID (`req_…`), identical across every chunk in one stream and returned in the `X-INFRO-Request-Id` response header. Use it with `GET /v1/requests/{id}`.
- `object` (string) — Always `chat.completion.chunk`.
- `created` (integer) — Unix timestamp, in seconds.
- `model` (string) — The model actually serving the response. With [fallbacks](https://infro.io/docs/features/fallbacks) configured, this can differ from the model you requested — read it from the stream rather than assuming.
- `provider` (string) — INFRO extension: the provider kind of your connection that is serving the response, for example `anthropic` or `aws-bedrock`.
- `connection` (string) — INFRO extension: the ID of your connection that is serving the response.
- `route / route_name` (string | null) — INFRO extensions: `route` is the position in the chain that served (`primary`, `standby_a`, …); `route_name` is the slug of the route you called, or `null` when `model` was a model ID.
- `choices[].delta` (object) — The incremental payload. `role` appears on the first chunk only; after that, `content` and/or `tool_calls` fragments.
- `choices[].finish_reason` (string | null) — `null` until the final chunk for the choice, where it is `stop`, `length`, `tool_calls`, or `content_filter`.
- `usage` (object) — Present only on the final usage chunk, and only when `stream_options.include_usage` is set.

### Keep-alive comments

INFRO does not currently send comment lines, and comment lines from the provider are not passed through. The SSE format allows them, though (lines starting with a colon, such as `: keep-alive`), so if you parse the stream by hand, skip any line that does not start with `data: `; the OpenAI SDKs handle this for you.

## Usage and cost

By default a stream carries no token counts. Set `stream_options: {"include_usage": true}` and INFRO appends one final chunk after the last content chunk: its `choices` array is empty, and its `usage` object includes `cost` — an estimate of what your provider will bill, with any [prompt caching](https://infro.io/docs/features/prompt-caching) discount the provider applied already reflected. It is the same figure a non-streaming response reports in `usage.cost`. `cost` is `null` when the model is unpriced or the provider reported no usage, and `usage.price_source` (`list`, `custom`, `custom_provider` or `provider`) says where the rate came from whenever a cost is present. The chunk also carries `prompt_tokens_details.cached_tokens`.

```text
data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481601,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[],"usage":{"prompt_tokens":24,"completion_tokens":118,"total_tokens":142,"prompt_tokens_details":{"cached_tokens":0},"cost":0.001228,"price_source":"list"}}

data: [DONE]
```

Because the usage chunk has an empty `choices` array, guard your delta access — code that unconditionally reads `chunk.choices[0]` breaks the moment you enable `include_usage`.

## Streaming tool calls

When the model calls a tool, the call arrives incrementally in `delta.tool_calls`. The first fragment for a call carries its `index`, `id`, and `function.name`; every later fragment carries the same `index` and a piece of `function.arguments`. Concatenate the argument fragments per index — the accumulated string is not valid JSON until the stream finishes with `finish_reason: "tool_calls"`.

```text
data: {"id":"req_7pL3sD9fH2jK5nBc","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.6-sol","route":"primary","route_name":null,"provider":"openai","connection":"conn_3Hq7wN2cL9dVe4Xa","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_bXk2","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]}

data: {"id":"req_7pL3sD9fH2jK5nBc","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.6-sol","route":"primary","route_name":null,"provider":"openai","connection":"conn_3Hq7wN2cL9dVe4Xa","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\": \"Ber"}}]},"finish_reason":null}]}

data: {"id":"req_7pL3sD9fH2jK5nBc","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.6-sol","route":"primary","route_name":null,"provider":"openai","connection":"conn_3Hq7wN2cL9dVe4Xa","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"lin\"}"}}]},"finish_reason":null}]}

data: {"id":"req_7pL3sD9fH2jK5nBc","object":"chat.completion.chunk","created":1787481701,"model":"openai/gpt-5.6-sol","route":"primary","route_name":null,"provider":"openai","connection":"conn_3Hq7wN2cL9dVe4Xa","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]
```

Parallel tool calls interleave in the same stream, distinguished by `index`. The format is identical for every model on INFRO — Anthropic, Gemini, and DeepSeek tool-call formats are normalized to it. See [Tool calling](https://infro.io/docs/features/tool-calling) for defining tools and returning results.

## Errors mid-stream

Provider failures before the first event never reach you directly: INFRO moves the request to the next target on the route, or the next of your connections that serves the model, and model-level [fallbacks](https://infro.io/docs/features/fallbacks) apply at the same point. If nothing can serve the request, you get a normal HTTP error status with a JSON body — see [Errors](https://infro.io/docs/api/errors).

Once tokens are flowing, re-routing is no longer possible. If the serving provider drops mid-generation, INFRO emits a final error event and closes the stream without `data: [DONE]`. A provider that goes silent for longer than the connection's timeout (never less than 120 seconds) ends the stream the same way, with type `request_timeout` and code `stream_stalled`.

```text
data: {"error": {"message": "The response stream ended before it completed.", "type": "upstream_error", "code": null}}
```

> **Warning:** Streams are not resumable. On a mid-stream error, retry the entire request and discard or replace the partial output — a retry regenerates from scratch and may produce different text.

Treat a mid-stream error event like a retryable HTTP status: retry with exponential backoff and jitter, the same policy as 408, 429, 502, and 503. Never auto-retry `invalid_request_error`, `authentication_error`, `budget_exceeded`, or `model_not_found`, or an error whose `code` is `guardrail_blocked` or `guardrail_unavailable` — those fail identically every time. An output [guardrail](https://infro.io/docs/features/guardrails) also holds back the last few characters of a stream before releasing them, and a `block` match ends the stream with that error event; text sent before it was already delivered.

## Consuming a stream

The OpenAI SDKs handle SSE parsing, comment lines, and the `[DONE]` sentinel. The examples below print content as it arrives and read the final usage chunk; note the guard on `choices`, which is empty on that last chunk. `usage.cost` is an INFRO extension, so typed SDKs need a loose read.

**Python**

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
)

stream = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",
    messages=[{"role": "user", "content": "Explain SSE in one paragraph."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:  # final chunk when include_usage is set
        cost = getattr(chunk.usage, "cost", None)  # INFRO extension
        print(
            f"\n\n{chunk.usage.prompt_tokens} in / "
            f"{chunk.usage.completion_tokens} out — ${cost}"
        )
```

**TypeScript**

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-5",
  messages: [{ role: "user", content: "Explain SSE in one paragraph." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) process.stdout.write(delta);

  if (chunk.usage) {
    // final chunk when include_usage is set; cost is an INFRO extension
    const cost = (chunk.usage as { cost?: number }).cost;
    console.log(
      `\n\n${chunk.usage.prompt_tokens} in / ` +
        `${chunk.usage.completion_tokens} out — $${cost}`
    );
  }
}
```

Streaming composes with everything else on the [chat completions](https://infro.io/docs/api/chat-completions) endpoint: [structured outputs](https://infro.io/docs/features/structured-outputs) stream as raw JSON text, and [routes](https://infro.io/docs/routes) and fallbacks are resolved before the stream opens, so they never affect the wire format.