# Chat completions

> Full reference for POST /v1/chat/completions: the OpenAI parameters it models plus INFRO's fallbacks, logging and metadata extensions.

Source: https://infro.io/docs/api/chat-completions

> **Note:** INFRO is in private early access. Running these examples requires an early-access account, an active trial or subscription entitlement, and your own provider connection. Self-serve purchasing is not generally open. See [Early access](https://infro.io/early-access).

The chat completions endpoint is the core of the API and is wire-compatible with OpenAI's. Any SDK or tool that speaks the OpenAI protocol works against it by changing the base URL.

```text
POST https://api.infro.io/v1/chat/completions
```

## Request body

- `model` (string, required) — A configured route slug, or a model ID in `vendor/model-name` form, e.g. `anthropic/claude-opus-5`. Use the model ids your connected providers serve; [`GET /v1/models`](https://infro.io/docs/models) lists what your key can call.
- `messages` (array, required) — Conversation history. Each message has `role` (`system` | `developer` | `user` | `assistant` | `tool`) and `content` (string, or an array of text/image parts for [vision](https://infro.io/docs/features/vision)).
- `stream` (boolean) — When `true`, tokens arrive as server-sent events. See [Streaming](https://infro.io/docs/api/streaming).
- `max_tokens` (integer) — Upper bound on generated tokens. When omitted, the provider's own default applies, with two exceptions: Anthropic requires a value, so INFRO sends the model's maximum output (32,768 tokens where the catalog lists none); and under a [blocking budget](https://infro.io/docs/org/spend-controls) INFRO sends that same ceiling to every provider, so a response cannot outrun what the budget reserved. `max_completion_tokens` is accepted as an alias. INFRO sends the ceiling in the field each model takes: OpenAI's reasoning models (GPT-5 and later, the o-series) refuse `max_tokens`, so they are sent `max_completion_tokens` whichever name you used.
- `temperature` (number) — Sampling temperature, `0`–`2`. Higher is more random. A model that refuses it is not sent it: Claude 4.7 and later (Opus 4.7 and 4.8, the Claude 5 family, Fable) refuse `temperature`, `top_p` and `top_k`, and OpenAI's o-series takes only its default. The response then lists what was dropped in `X-INFRO-Params-Dropped` (see below).
- `top_p` (number) — Nucleus sampling. Use temperature or top_p, not both: Claude 4.5 and 4.6 refuse the pair, so INFRO keeps `temperature` and drops `top_p` for them.
- `stop` (string | array) — Up to 4 sequences where generation halts.
- `frequency_penalty / presence_penalty` (number) — `-2.0`–`2.0`. Forwarded to the provider; whether it is honoured depends on the provider.
- `tools` (array) — Function definitions the model may call. Translated to each provider's own format — see [Tool calling](https://infro.io/docs/features/tool-calling).
- `tool_choice` (string | object) — `"auto"` (default), `"none"`, `"required"`, or `{"type": "function", "function": {"name": "..."}}` to force one tool.
- `response_format` (object) — `{"type": "json_object"}` for JSON mode or `{"type": "json_schema", "json_schema": {...}}` for validated output. See [Structured outputs](https://infro.io/docs/features/structured-outputs).
- `seed` (integer) — Best-effort deterministic sampling on providers that support it.
- `user` (string) — Opaque end-user ID, forwarded to the provider as sent. To attribute cost to your users in INFRO's request log, send `metadata.user_id`.
- `stream_options` (object) — `{"include_usage": true}` appends a final chunk with token counts when streaming.

## INFRO extensions

INFRO-specific fields sit at the top level of the request body (use `extra_body` in the OpenAI SDKs). Other OpenAI-compatible backends ignore them, so requests stay portable.

- `routing` (object) — `routing.providers` chooses which of your providers may serve this request, by provider id (`openai`, `aws-bedrock`, `openrouter`): `{"allow": ["aws-bedrock"]}` keeps only those, `{"deny": ["openrouter"]}` removes those. It applies to the model's connections, a route's targets and the fallbacks, and narrows what your API key allows without widening it. Leave it out to let the gateway choose among every connection that serves the model. An id that is not a provider is `400` `unknown_provider`; when nothing it allows can serve, the `503` `connection_out_of_scope` says so. Legacy: `routing.policy` (`cheapest`, `fastest`, `balanced`) is accepted and ignored. `routing.regions` is checked against your organization's region policy and recorded, but does not move a request between your connections. Use a named [route](https://infro.io/docs/routes) to set the order connections are tried in.
- `fallbacks` (array) — Up to 5 ordered model IDs to try after every connection for `model` has failed, e.g. `["openai/gpt-5.6-terra"]`. An ID that is not a model you can call is refused with `400` code `unknown_fallback_model`, naming it; a known model is skipped when none of your connections serves it, when your policy does not allow it, or when it is a different modality. Ignored when `model` is a route slug (an unknown ID is still refused): a route's failover is set on the route. See [Failover & fallbacks](https://infro.io/docs/features/fallbacks).
- `logging` (boolean) — Set `false` to disable optional prompt and response capture. Organization opt-in and plan limits also apply. Usage metadata is still recorded for platform metering and estimated provider cost. See [Privacy & data](https://infro.io/docs/privacy).
- `metadata` (object) — Your own annotation, recorded on the request and never sent to the provider. `user_id`, `session_id` and `tags.<key>` can be filtered in the request log.

### Response cache

The [response cache](https://infro.io/docs/features/caching) extension. It narrows what the organization's settings allow and never turns the cache on where they have it off; the headers `X-INFRO-Cache`, `X-INFRO-Cache-TTL` and `X-INFRO-Cache-Namespace` are equivalents, and the body wins field by field. `logging: false` also means the answer is never written to the cache.

- `cache` (object) — `{mode, ttl_seconds, namespace}`. `mode`: `default`, `bypass`, `refresh` or `read_only`. `ttl_seconds`: `60`–`2592000` and at most the plan's maximum, otherwise `400` `cache_ttl_too_long`. `namespace`: 1–128 characters of letters, digits and `. _ : -`, partitioning the key. A malformed header is `400` `invalid_cache_control`.

## Example request

**curl**

```bash
curl https://api.infro.io/v1/chat/completions \
  -H "Authorization: Bearer $INFRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Summarize SSE in one sentence."}
    ],
    "max_tokens": 200,
    "fallbacks": ["openai/gpt-5.6-terra"]
  }'
```

**Python**

```python
completion = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "Summarize SSE in one sentence."},
    ],
    max_tokens=200,
    extra_body={
        "fallbacks": ["openai/gpt-5.6-terra"],
    },
)
```

**TypeScript**

```typescript
const completion = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-5",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "Summarize SSE in one sentence." },
  ],
  max_tokens: 200,
  // @ts-expect-error INFRO extensions
  fallbacks: ["openai/gpt-5.6-terra"],
});
```

## Response

```json
{
  "id": "req_9f27c1d0a4b84e6f",
  "object": "chat.completion",
  "created": 1755950000,
  "model": "anthropic/claude-sonnet-5",
  "route": "primary",
  "route_name": null,
  "provider": "anthropic",
  "connection": "conn_4b1e9a07",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Server-sent events stream one-way updates from server to client over a single HTTP connection."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 31,
    "completion_tokens": 19,
    "total_tokens": 50,
    "cost": 0.000378,
    "price_source": "list"
  }
}
```

- `route` (string) — INFRO extension: which position in the resolved chain served the request — `primary`, or `standby_a`, `standby_b` and so on after a failover. It tells you whether your traffic moved between your connections. The top-level `id` is INFRO's request id, the same value as the `X-INFRO-Request-Id` header.
- `route_name` (string | null) — INFRO extension: the slug of the named [route](https://infro.io/docs/routes) the request called, or `null` when it named a model directly.
- `provider` (string) — INFRO extension: the provider kind whose account served the request — `openai`, `anthropic`, `aws-bedrock` and so on. It is one of your own connections; `connection` names which.
- `connection` (string) — INFRO extension: the id of the provider connection that served the request.
- `usage.cost` (number) — INFRO extension: what your provider will bill you for this request, in USD. An **estimate**, unless the provider reported the cost in its own response (`price_source` is then `provider`, and the provider's figure is also in `usage.cost_usd`). INFRO takes no part of it. `null` when the model is unpriced; that does not mean free.
- `usage.price_source` (string) — INFRO extension: where `cost` came from — `provider` (reported by the provider in its response), `custom_provider` (your negotiated rate for the provider that served it), `custom` (your rate for the model, whatever serves it) or `list` (the model vendor's published rate). Absent when `cost` is `null`. See [billing](https://infro.io/docs/org/billing).
- `guardrails` (object) — INFRO extension, present only when your organization's [guardrails](https://infro.io/docs/features/guardrails) redacted, flagged or blocked something: `{action, findings}`, where each finding is `{detector, stage, action, count}`. Counts only — the matched text is never returned. On a stream it is on each choice's finish chunk and on the usage chunk.
- `finish_reason` (string) — `stop`, `length` (hit `max_tokens`), `tool_calls`, or `content_filter`.

> **Warning:** Always branch on `finish_reason` before trusting output: `length` means the response was truncated mid-thought, and `tool_calls` means `message.tool_calls` is populated and `content` may be empty.

When the model that served refuses a sampling parameter you sent, INFRO does not send it and the response carries `X-INFRO-Params-Dropped` naming each one, for example `X-INFRO-Params-Dropped: temperature, top_p`. Only sampling parameters a model takes at its default and nothing else are ever dropped (`temperature`, `top_p`, `top_k`, `frequency_penalty`, `presence_penalty`, `seed`, `logprobs`, `top_logprobs`, `logit_bias`); a refused `stop`, `tools` or `response_format` is returned to you as the provider's `400`. When a provider refuses a parameter INFRO has no rule for yet, INFRO drops it and asks the same connection once more, and remembers the answer for the following requests. The header is absent when nothing was dropped.

When a guardrail acted on the request, the response also carries the header `X-INFRO-Guardrails: redacted`, `flagged` or `blocked`. On a stream the header reflects the input check only, because it is sent before any response text exists. A request refused by a guardrail returns `403` with `code: "guardrail_blocked"` or `"guardrail_unavailable"` — see [Guardrails](https://infro.io/docs/features/guardrails).

### Response cache fields

Every chat response that reached the cache step carries `X-INFRO-Cache: HIT | MISS | BYPASS | DISABLED | UNCACHEABLE`; a hit adds `X-INFRO-Cache-Age` and `X-INFRO-Cache-Source`. The body gains a top-level `cache` object, absent when the cache is off for the request: `{status}` with `miss`, `bypass` or `uncacheable`, or on a hit `{status: "hit", age_seconds, source_request_id, provider_billed: false}`. On a hit, `usage` carries the original token counts, `usage.cost` is `0` and `usage.price_source` is `gateway_cache`; `model`, `provider`, `connection` and `route` describe what produced the answer originally, and `route_name` is the route this request named. See [Response cache](https://infro.io/docs/features/caching#what-the-response-says).

## Compatibility notes

- Top-level parameters INFRO does not model are forwarded to the provider unchanged, so a new OpenAI parameter does not break your requests. Whether the provider honours one is up to the provider; a sampling parameter the serving model refuses is dropped and reported in `X-INFRO-Params-Dropped`.
- `usage` token counts are the counts your provider reported for the request; INFRO does not re-tokenize.
- The legacy `POST /v1/completions` endpoint is not supported; use chat completions.