# From direct OpenAI calls to a gateway: a canary runbook

URL: https://infro.io/blog/migrate-direct-openai-to-gateway-canary
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Architecture and migration
Format: Guide

> Treat the move to a gateway as a canary: first confirm every OpenAI endpoint you call is served, then send a small, stable cohort of users through the gateway while everyone else stays direct, compare the two populations on errors, 429s, latency and output shape, and advance only when they match. Rollback is setting the gateway share back to zero.

Key takeaways:
- Inventory the endpoints you call before touching traffic; an endpoint the gateway does not serve has to stay on the direct path.
- Split by a stable cohort such as a hashed user id, so one user never flips between paths mid-conversation.
- Compare the canary with the control population at the same time, not with last week's numbers.
- Both paths draw on the same OpenAI account and rate limit, so a 429 during the canary is shared, not caused by the gateway.
- Rollback is one configuration value. Keep the direct path wired until the soak period ends.

Most teams start by calling OpenAI directly: one SDK, one key in an environment variable, one base URL nobody thinks about. Putting a gateway in front of that is a small code change and a large operational one. Every model call in the product now takes a different network path, is authenticated by a different key, and is answered by a system that can fail over, refuse on policy and record what happened.

That is exactly the kind of change a canary exists for. The Google SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation"[1]. Here the change is not your code. It is the path your code's requests take.

## What actually changes when you add a gateway

For an application already using the OpenAI SDK, three settings change and everything else should not:

- **Base URL.** The client points at the gateway instead of `api.openai.com`.
- **Key.** The application holds a gateway key. The OpenAI key moves into the gateway's connection, where it is sealed.
- **Model id.** A gateway that serves several vendors names models with a vendor prefix: `openai/gpt-5.6-sol` rather than `gpt-5.6-sol`.

The request body, streaming format and error shape stay OpenAI-compatible. What changes under them is the failure surface: a new network hop, a new set of refusals (budgets, allowlists, a missing connection), and a response that names which account served it.

[Diagram: The canary split lives in your application. The control cohort keeps calling OpenAI directly; the canary cohort sends the same request to the gateway, which calls the same OpenAI account with your own credential.]

## Step 1: inventory the endpoints you call

Before any traffic moves, list every OpenAI API your code calls, not just the main one. Grep for the SDK's resource names: `chat.completions`, `responses`, `embeddings`, `images`, `audio`, `files`, `batches`. A gateway serves a subset of any provider's surface, and an unserved endpoint is a hard failure on the canary path, not a subtle one.

[Diagram: Sort every call site into one column before the first user moves. The right-hand column stays on the direct client until the gateway serves it.]

Two cases need a decision before starting:

- **The Responses API.** Code written against `client.responses.create` has to move to chat completions before it can use a chat-completions gateway. That is a code migration with its own evaluation, and it should ship and settle on the direct path first, so the canary tests one change at a time.
- **Stateful APIs** such as files and batches. Keep a second, direct client for them. A gateway migration does not have to be all-or-nothing per SDK; it has to be all-or-nothing per call site.

## Step 2: build the switch into the client

The split belongs in one place: the function that constructs your client. It needs a stable cohort per user, so a user does not bounce between paths mid-conversation, and a share you can change without a deploy. Fowler describes this as a canary cohort created by "a random sampling of 1% of the user base", consistently routed by user[2]. Operational switches like this one need to change fast; waiting for a release to flip one defeats the purpose[2].

```python
import hashlib
import os

from openai import OpenAI

_direct = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    timeout=60.0,
    max_retries=2,
)
_gateway = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
    timeout=60.0,
    max_retries=1,
)


def gateway_share() -> float:
    # Read from your flag service or config store, not from code.
    return float(os.environ.get("GATEWAY_SHARE", "0"))


def in_canary(user_id: str) -> bool:
    bucket = int(hashlib.sha256(user_id.encode()).hexdigest()[:8], 16) % 10_000
    return bucket < gateway_share() * 10_000


def client_for(user_id: str) -> tuple[OpenAI, str, str]:
    """Return the client, the model id for that path, and the cohort label."""
    if in_canary(user_id):
        return _gateway, "openai/gpt-5.6-sol", "canary"
    return _direct, "gpt-5.6-sol", "control"
```

```python
client, model, cohort = client_for(user.id)
extra = {"metadata": {"feature": "support-reply"}} if cohort == "canary" else {}

response = client.chat.completions.create(
    model=model,
    messages=messages,
    stream=False,
    extra_body=extra,
)
metrics.increment("llm.requests", tags={"cohort": cohort})
```

The same shape in TypeScript, where the SDK takes `baseURL`:

```typescript
import OpenAI from "openai";
import { createHash } from "node:crypto";

const direct = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, timeout: 60_000, maxRetries: 2 });
const gateway = new OpenAI({
  baseURL: "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
  timeout: 60_000,
  maxRetries: 1,
});

export function clientFor(userId: string, share: number) {
  const bucket = parseInt(createHash("sha256").update(userId).digest("hex").slice(0, 8), 16) % 10_000;
  return bucket < share * 10_000
    ? { client: gateway, model: "openai/gpt-5.6-sol", cohort: "canary" as const }
    : { client: direct, model: "gpt-5.6-sol", cohort: "control" as const };
}
```

Three details in that code are deliberate. The timeout is explicit, because the Python SDK's default is ten minutes[3], far longer than any user will wait. The gateway client retries less, because the gateway already moves a failed attempt to another connection before the first token. And the cohort label is emitted on every metric, because the whole method depends on comparing the two populations.

The SDK also reads `OPENAI_BASE_URL` from the environment[3]. That is convenient for a full cut-over and wrong for a canary: it moves every request in the process at once.

## Step 3: compare populations, not days

The workbook is specific about evaluation: compare the canary against a control population running at the same time, because before-and-after comparisons pick up every other change in the system[1]. It also shows why aggregate dashboards hide canary problems: a 5% canary with a 20% error rate adds only one point to the overall error rate[1]. Break every signal down by cohort.

[Diagram: What to compare between cohorts, and where each number comes from. The request records supply the gateway side; your application metrics supply both.]

Two comparisons deserve care:

- **Latency.** The gateway adds a network hop. Decide before you start how much extra time to first token is acceptable, then measure it from the client in both cohorts rather than trusting either side's server timings alone.
- **Output shape.** The model is the same, so `finish_reason` distribution and tokens per request should match. A shift means something in the request changed on the way, such as a default `max_tokens`, and is worth finding before it reaches every user.

## Step 4: advance in stages

Each stage should last long enough to include a daily peak, and only one change should be in flight at a time, since overlapping canaries make it impossible to say which one caused a signal[1].

[Diagram: An example progression. Illustrative stage sizes; the right durations depend on your traffic volume and how quickly a problem would show.]

> **Example: A stage gate written down in advance**
> Before stage 2 (5%): canary error rate within 0.2 points of control over 24 hours; p95 time to first token no more than 150 ms above control; no new `finish_reason` values; zero parse failures in the structured-output validator; no INFRO refusals (402, 403) in the request log. These thresholds are illustrative. The point is that they are decided before the numbers arrive.

## Step 5: roll back without a deploy

Rollback is setting the share to zero. Because the cohort is computed per request from a configuration value, the next request from every user goes direct again. Nothing needs to be migrated back: there is no state on the gateway path that the direct path depends on.

[Diagram: Most canary problems fall into one of four shapes. Only the first means rolling back; configuration and policy refusals are fixed in place.]

Keep the rollback path real until the end:

1. Leave the direct client and `OPENAI_API_KEY` in place through a soak period at 100%.
2. Remove the direct call sites in one change, after the soak.
3. Then revoke the OpenAI key the application held, so the only copy of an OpenAI credential is the one sealed in the gateway connection.

## Running the canary against INFRO

In INFRO the gateway path calls your own OpenAI account: you connect an OpenAI API key under Providers, it is sealed and never returned, and your provider bills you directly ([connect a provider](https://infro.io/docs/providers)). INFRO checks the key when it is saved, so a typo shows up there rather than in the first canary request.

A few behaviours shape the runbook:

- **Model ids.** `openai/gpt-5.6-sol` resolves to every enabled, healthy connection that serves OpenAI models, in creation order ([connect a provider](https://infro.io/docs/providers)). The response's `model` field carries that catalog id, so any code comparing `response.model` with a bare name needs updating.
- **Extra fields.** Responses add `provider`, `connection`, `route` and `usage.cost`, an estimate of what OpenAI will bill for the request, `null` when unpriced ([chat completions](https://infro.io/docs/api/chat-completions)). The top-level `id` equals the `X-INFRO-Request-Id` header, which is the id to log beside your own.
- **Environments fail closed.** A key that names no environment cannot reach a connection scoped to `production`. Mismatches return `503 connection_out_of_scope`, not a provider error ([connect a provider](https://infro.io/docs/providers)).
- **Refusals that never reach OpenAI.** Budget, subscription and allowlist refusals return 402 or 403 immediately without failover ([fallbacks](https://infro.io/docs/features/fallbacks)). They are policy working, and they should be expected in the canary before they surprise production.
- **Shared limits.** Both cohorts spend the same OpenAI rate limit. A provider 429 returns `429 provider_rate_limited` with OpenAI's `Retry-After`, and it does not count against the connection's health ([connect a provider](https://infro.io/docs/providers)).

To read the canary side, filter the request log by feature and outcome:

```bash
curl "https://api.infro.io/v1/requests?feature=support-reply&status=error&limit=50" \
  -H "Authorization: Bearer $INFRO_API_KEY"
```

Each record carries `ttft_ms`, `finish_reason`, token counts and every fallback attempt, and the key's history access decides which requests it can read ([request traces](https://infro.io/docs/observability/traces)).

After the migration, the next change is easier. A [route](https://infro.io/docs/routes) with the `weighted` strategy picks the first target in proportion to its weight and still fails over to the rest, so a second provider can be canaried with a weight change instead of a code change. With the management API that is one versioned write:

```bash
curl https://api.infro.io/manage/v1/routes/rt_01J9Z6 \
  -X PATCH \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H 'If-Match: "4"' \
  -H "Content-Type: application/json" \
  -d '{
    "strategy": "weighted",
    "targets": [
      {"connection_id": "conn_openai", "model": "openai/gpt-5.6-sol", "weight": 95},
      {"connection_id": "conn_azure", "model": "openai/gpt-5.6-sol", "weight": 5}
    ]
  }'
```

`targets` replaces the whole list, and `If-Match` refuses the write if someone changed the route since you read it ([management API resources](https://infro.io/docs/manage/resources)).

## Checklist

- [ ] Every call site sorted into gateway or direct, with the Responses API and stateful APIs kept direct.
- [ ] Client factory with a hashed, stable cohort and a share read from configuration.
- [ ] Explicit timeouts on both clients; fewer SDK retries on the gateway client.
- [ ] Every metric tagged with the cohort; every gateway request tagged with a feature.
- [ ] Gate thresholds written down before stage 1.
- [ ] Rollback tested once at stage 0 by setting the share to zero.
- [ ] Direct path kept through the soak; OpenAI key removed from the application afterwards.

## Sources

1. Canarying Releases — The Site Reliability Workbook — Google. https://sre.google/workbook/canarying-releases/ (accessed 2026-09-29)
2. Feature Toggles (aka Feature Flags) — Martin Fowler. https://martinfowler.com/articles/feature-toggles.html (accessed 2026-09-29)
3. openai-python README — OpenAI. https://github.com/openai/openai-python (accessed 2026-09-29)