Streaming

Set stream: true on any chat completions request and INFRO returns the response as a stream of server-sent events (SSE) instead of a single JSON body. Tokens arrive as they are generated, so you can render output immediately.

The wire format is OpenAI-compatible: chat.completion.chunk events with incremental delta payloads, terminated by data: [DONE]. Any OpenAI SDK consumes it unchanged — point the client at https://api.infro.io/v1 as shown in Quickstart. This page covers the raw protocol for anyone parsing SSE by hand, plus INFRO-specific details: the cost estimate in the final usage chunk, incremental tool-call arguments, and mid-stream error behavior.

Requesting a stream

Two request fields control streaming. Everything else — routing, fallbacks, tools, response formats — works exactly as it does without streaming.

streamboolean

Set to true to receive the response as an SSE stream of chat.completion.chunk events. Defaults to false.

stream_optionsobject

Streaming-only options. Ignored unless stream is true.

stream_options.include_usageboolean

When true, INFRO sends one extra chunk before data: [DONE] whose usage object carries prompt_tokens, completion_tokens, and cost — an estimate of what your provider will bill. That chunk's choices array is empty.

curl https://api.infro.io/v1/chat/completions \
  -N \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [{"role": "user", "content": "Explain SSE in one paragraph."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

The -N flag disables curl's output buffering so events print as they arrive. The response has Content-Type: text/event-stream and carries the standard X-RateLimit-* headers described in Limits.

Reading the event stream

Each event is a single line of the form data: {json}, separated by blank lines. The first chunk sets delta.role to assistant; later chunks carry delta.content fragments; the final chunk for a choice has an empty delta and a non-null finish_reason. A successful stream ends with the literal sentinel data: [DONE] — it is not JSON, so match it before parsing. After a mid-stream error event the connection closes without the sentinel, so treat connection close as terminal too (see Errors).

data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[{"index":0,"delta":{"content":"Server"},"finish_reason":null}]}

data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481600,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[{"index":0,"delta":{"content":"-sent events are..."},"finish_reason":null}]}

data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481601,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

A single chunk, formatted for readability:

{
  "id": "req_2mV9kQ7xT4bN8wRz",
  "object": "chat.completion.chunk",
  "created": 1787481600,
  "model": "anthropic/claude-sonnet-5",
  "route": "primary",
  "route_name": null,
  "provider": "anthropic",
  "connection": "conn_8Kx2mQ4vR7tYb1Zp",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": "-sent events are..."
      },
      "finish_reason": null
    }
  ]
}

idstring

The INFRO request ID (req_…), identical across every chunk in one stream and returned in the X-INFRO-Request-Id response header. Use it with GET /v1/requests/{id}.

objectstring

Always chat.completion.chunk.

createdinteger

Unix timestamp, in seconds.

modelstring

The model actually serving the response. With fallbacks configured, this can differ from the model you requested — read it from the stream rather than assuming.

providerstring

INFRO extension: the provider kind of your connection that is serving the response, for example anthropic or aws-bedrock.

connectionstring

INFRO extension: the ID of your connection that is serving the response.

route / route_namestring | null

INFRO extensions: route is the position in the chain that served (primary, standby_a, …); route_name is the slug of the route you called, or null when model was a model ID.

choices[].deltaobject

The incremental payload. role appears on the first chunk only; after that, content and/or tool_calls fragments.

choices[].finish_reasonstring | null

null until the final chunk for the choice, where it is stop, length, tool_calls, or content_filter.

usageobject

Present only on the final usage chunk, and only when stream_options.include_usage is set.

Keep-alive comments

INFRO does not currently send comment lines, and comment lines from the provider are not passed through. The SSE format allows them, though (lines starting with a colon, such as : keep-alive), so if you parse the stream by hand, skip any line that does not start with data: ; the OpenAI SDKs handle this for you.

Usage and cost

By default a stream carries no token counts. Set stream_options: {"include_usage": true} and INFRO appends one final chunk after the last content chunk: its choices array is empty, and its usage object includes cost — an estimate of what your provider will bill, with any prompt caching discount the provider applied already reflected. It is the same figure a non-streaming response reports in usage.cost. cost is null when the model is unpriced or the provider reported no usage, and usage.price_source (list, custom, custom_provider or provider) says where the rate came from whenever a cost is present. The chunk also carries prompt_tokens_details.cached_tokens.

data: {"id":"req_2mV9kQ7xT4bN8wRz","object":"chat.completion.chunk","created":1787481601,"model":"anthropic/claude-sonnet-5","route":"primary","route_name":null,"provider":"anthropic","connection":"conn_8Kx2mQ4vR7tYb1Zp","choices":[],"usage":{"prompt_tokens":24,"completion_tokens":118,"total_tokens":142,"prompt_tokens_details":{"cached_tokens":0},"cost":0.001228,"price_source":"list"}}

data: [DONE]

Because the usage chunk has an empty choices array, guard your delta access — code that unconditionally reads chunk.choices[0] breaks the moment you enable include_usage.

Streaming tool calls

When the model calls a tool, the call arrives incrementally in delta.tool_calls. The first fragment for a call carries its index, id, and function.name; every later fragment carries the same index and a piece of function.arguments. Concatenate the argument fragments per index — the accumulated string is not valid JSON until the stream finishes with finish_reason: "tool_calls".

data: {"id":"req_7pL3sD9fH2jK5nBc","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.6-sol","route":"primary","route_name":null,"provider":"openai","connection":"conn_3Hq7wN2cL9dVe4Xa","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_bXk2","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]}

data: {"id":"req_7pL3sD9fH2jK5nBc","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.6-sol","route":"primary","route_name":null,"provider":"openai","connection":"conn_3Hq7wN2cL9dVe4Xa","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\": \"Ber"}}]},"finish_reason":null}]}

data: {"id":"req_7pL3sD9fH2jK5nBc","object":"chat.completion.chunk","created":1787481700,"model":"openai/gpt-5.6-sol","route":"primary","route_name":null,"provider":"openai","connection":"conn_3Hq7wN2cL9dVe4Xa","choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"lin\"}"}}]},"finish_reason":null}]}

data: {"id":"req_7pL3sD9fH2jK5nBc","object":"chat.completion.chunk","created":1787481701,"model":"openai/gpt-5.6-sol","route":"primary","route_name":null,"provider":"openai","connection":"conn_3Hq7wN2cL9dVe4Xa","choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

data: [DONE]

Parallel tool calls interleave in the same stream, distinguished by index. The format is identical for every model on INFRO — Anthropic, Gemini, and DeepSeek tool-call formats are normalized to it. See Tool calling for defining tools and returning results.

Errors mid-stream

Provider failures before the first event never reach you directly: INFRO moves the request to the next target on the route, or the next of your connections that serves the model, and model-level fallbacks apply at the same point. If nothing can serve the request, you get a normal HTTP error status with a JSON body — see Errors.

Once tokens are flowing, re-routing is no longer possible. If the serving provider drops mid-generation, INFRO emits a final error event and closes the stream without data: [DONE]. A provider that goes silent for longer than the connection's timeout (never less than 120 seconds) ends the stream the same way, with type request_timeout and code stream_stalled.

data: {"error": {"message": "The response stream ended before it completed.", "type": "upstream_error", "code": null}}

Treat a mid-stream error event like a retryable HTTP status: retry with exponential backoff and jitter, the same policy as 408, 429, 502, and 503. Never auto-retry invalid_request_error, authentication_error, budget_exceeded, or model_not_found, or an error whose code is guardrail_blocked or guardrail_unavailable — those fail identically every time. An output guardrail also holds back the last few characters of a stream before releasing them, and a block match ends the stream with that error event; text sent before it was already delivered.

Consuming a stream

The OpenAI SDKs handle SSE parsing, comment lines, and the [DONE] sentinel. The examples below print content as it arrives and read the final usage chunk; note the guard on choices, which is empty on that last chunk. usage.cost is an INFRO extension, so typed SDKs need a loose read.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
)

stream = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",
    messages=[{"role": "user", "content": "Explain SSE in one paragraph."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:  # final chunk when include_usage is set
        cost = getattr(chunk.usage, "cost", None)  # INFRO extension
        print(
            f"\n\n{chunk.usage.prompt_tokens} in / "
            f"{chunk.usage.completion_tokens} out — ${cost}"
        )

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-5",
  messages: [{ role: "user", content: "Explain SSE in one paragraph." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) process.stdout.write(delta);

  if (chunk.usage) {
    // final chunk when include_usage is set; cost is an INFRO extension
    const cost = (chunk.usage as { cost?: number }).cost;
    console.log(
      `\n\n${chunk.usage.prompt_tokens} in / ` +
        `${chunk.usage.completion_tokens} out — $${cost}`
    );
  }
}

Streaming composes with everything else on the chat completions endpoint: structured outputs stream as raw JSON text, and routes and fallbacks are resolved before the stream opens, so they never affect the wire format.