Streaming

Set stream: true on any chat completions request and INFRO returns the response as a stream of server-sent events (SSE) instead of a single JSON body. Tokens arrive as they are generated, so you can render output immediately.

The wire format is OpenAI-compatible: chat.completion.chunk events with incremental delta payloads, terminated by data: [DONE]. Any OpenAI SDK consumes it unchanged — point the client at https://api.infro.io/v1 as shown in Quickstart. This page covers the raw protocol for anyone parsing SSE by hand, plus INFRO-specific details: exact cost in the final usage chunk, incremental tool-call arguments, and mid-stream error behavior.

Requesting a stream

Two request fields control streaming. Everything else — routing, fallbacks, tools, response formats — works exactly as it does without streaming.

streamboolean

Set to true to receive the response as an SSE stream of chat.completion.chunk events. Defaults to false.

stream_optionsobject

Streaming-only options. Ignored unless stream is true.

stream_options.include_usageboolean

When true, INFRO sends one extra chunk before data: [DONE] whose usage object carries prompt_tokens, completion_tokens, and cost — the exact USD charged. That chunk's choices array is empty.

curl https://api.infro.io/v1/chat/completions \
  -N \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{"role": "user", "content": "Explain SSE in one paragraph."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

The -N flag disables curl's output buffering so events print as they arrive. The response has Content-Type: text/event-stream and carries the standard X-RateLimit-* headers described in Limits.

Reading the event stream

Each event is a single line of the form data: {json}, separated by blank lines. The first chunk sets delta.role to assistant; later chunks carry delta.content fragments; the final chunk for a choice has an empty delta and a non-null finish_reason. A successful stream ends with the literal sentinel data: [DONE] — it is not JSON, so match it before parsing. After a mid-stream error event the connection closes without the sentinel, so treat connection close as terminal too (see Errors).

data: {"id":"chatcmpl-8a4e2f","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","provider":"anthropic","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-8a4e2f","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","provider":"anthropic","choices":[{"index":0,"delta":{"content":"Server"},"finish_reason":null}]}

data: {"id":"chatcmpl-8a4e2f","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","provider":"anthropic","choices":[{"index":0,"delta":{"content":"-sent events are..."},"finish_reason":null}]}

data: {"id":"chatcmpl-8a4e2f","object":"chat.completion.chunk","created":1787481601,"model":"anthropic/claude-sonnet-5","provider":"anthropic","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

A single chunk, formatted for readability:

{
  "id": "chatcmpl-8a4e2f",
  "object": "chat.completion.chunk",
  "created": 1787481600,
  "model": "anthropic/claude-sonnet-5",
  "provider": "anthropic",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "-sent events are..."
      },
      "finish_reason": null
    }
  ]
}

idstring

Completion id. Identical across every chunk in one stream.

objectstring

Always chat.completion.chunk.

createdinteger

Unix timestamp, in seconds.

modelstring

The model actually serving the response. With fallbacks configured, this can differ from the model you requested — read it from the stream rather than assuming.

providerstring

INFRO extension: which provider is serving the response.

choices[].deltaobject

The incremental payload. role appears on the first chunk only; after that, content and/or tool_calls fragments.

choices[].finish_reasonstring | null

null until the final chunk for the choice, where it is stop, length, tool_calls, or content_filter.

usageobject

Present only on the final usage chunk, and only when stream_options.include_usage is set.

Keep-alive comments

An SSE stream can also contain comment lines — lines starting with a colon, such as : keep-alive — used to stop proxies and load balancers from closing an idle connection while a request is queued. They carry no data. If you parse the stream by hand, skip any line that does not start with data: ; the OpenAI SDKs handle this for you.

Usage and cost

By default a stream carries no token counts. Set stream_options: {"include_usage": true} and INFRO appends one final chunk after the last content chunk: its choices array is empty, and its usage object includes cost — the exact USD charged, with any prompt caching discounts already applied. It is the same figure a non-streaming response reports in usage.cost.

data: {"id":"chatcmpl-8a4e2f","object":"chat.completion.chunk","created":1787481601,"model":"anthropic/claude-sonnet-5","provider":"anthropic","choices":[],"usage":{"prompt_tokens":24,"completion_tokens":118,"total_tokens":142,"cost":0.001842}}

data: [DONE]

Because the usage chunk has an empty choices array, guard your delta access — code that unconditionally reads chunk.choices[0] breaks the moment you enable include_usage.

Streaming tool calls

When the model calls a tool, the call arrives incrementally in delta.tool_calls. The first fragment for a call carries its index, id, and function.name; every later fragment carries the same index and a piece of function.arguments. Concatenate the argument fragments per index — the accumulated string is not valid JSON until the stream finishes with finish_reason: "tool_calls".

data: {"id":"chatcmpl-1f9b","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.1","provider":"openai","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_bXk2","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-1f9b","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.1","provider":"openai","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\": \"Ber"}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-1f9b","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.1","provider":"openai","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"lin\"}"}}]},"finish_reason":null}]}

data: {"id":"chatcmpl-1f9b","object":"chat.completion.chunk","created":1787481701,"model":"openai/gpt-5.1","provider":"openai","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]

Parallel tool calls interleave in the same stream, distinguished by index. The format is identical for every model on INFRO — Anthropic, Gemini, and DeepSeek tool-call formats are normalized to it. See Tool calling for defining tools and returning results.

Errors mid-stream

Provider failures before the first token never reach you: INFRO re-routes 429s, 5xx errors, and timeouts to another provider automatically, and model-level fallbacks apply at the same point. If nothing can serve the request, you get a normal HTTP error status with a JSON body — see Errors.

Once tokens are flowing, re-routing is no longer possible. If the serving provider drops mid-generation, INFRO emits a final error event and closes the stream.

data: {"error": {"message": "Provider disconnected while streaming", "type": "upstream_error", "code": null}}

Treat a mid-stream error event like a retryable HTTP status: retry with exponential backoff and jitter, the same policy as 408, 429, 502, and 503. Never auto-retry invalid_request_error, authentication_error, insufficient_credits, or model_not_found — those fail identically every time.

Consuming a stream

The OpenAI SDKs handle SSE parsing, comment lines, and the [DONE] sentinel. The examples below print content as it arrives and read the final usage chunk; note the guard on choices, which is empty on that last chunk. usage.cost is an INFRO extension, so typed SDKs need a loose read.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
)

stream = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",
    messages=[{"role": "user", "content": "Explain SSE in one paragraph."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:  # final chunk when include_usage is set
        cost = getattr(chunk.usage, "cost", None)  # INFRO extension
        print(
            f"\n\n{chunk.usage.prompt_tokens} in / "
            f"{chunk.usage.completion_tokens} out — ${cost}"
        )

Streaming composes with everything else on the chat completions endpoint: structured outputs stream as raw JSON text, and routing policies apply before the stream opens, so they never affect the wire format.