From direct OpenAI calls to a gateway: a canary runbook

Most teams start by calling OpenAI directly: one SDK, one key in an environment variable, one base URL nobody thinks about. Putting a gateway in front of that is a small code change and a large operational one. Every model call in the product now takes a different network path, is authenticated by a different key, and is answered by a system that can fail over, refuse on policy and record what happened.

That is exactly the kind of change a canary exists for. The Google SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation"¹. Here the change is not your code. It is the path your code's requests take.

What actually changes when you add a gateway

For an application already using the OpenAI SDK, three settings change and everything else should not:

• Base URL. The client points at the gateway instead of api.openai.com.
• Key. The application holds a gateway key. The OpenAI key moves into the gateway's connection, where it is sealed.
• Model id. A gateway that serves several vendors names models with a vendor prefix: openai/gpt-5.6-sol rather than gpt-5.6-sol.

The request body, streaming format and error shape stay OpenAI-compatible. What changes under them is the failure surface: a new network hop, a new set of refusals (budgets, allowlists, a missing connection), and a response that names which account served it.

Step 1: inventory the endpoints you call

Before any traffic moves, list every OpenAI API your code calls, not just the main one. Grep for the SDK's resource names: chat.completions, responses, embeddings, images, audio, files, batches. A gateway serves a subset of any provider's surface, and an unserved endpoint is a hard failure on the canary path, not a subtle one.

Two cases need a decision before starting:

• The Responses API. Code written against client.responses.create has to move to chat completions before it can use a chat-completions gateway. That is a code migration with its own evaluation, and it should ship and settle on the direct path first, so the canary tests one change at a time.
• Stateful APIs such as files and batches. Keep a second, direct client for them. A gateway migration does not have to be all-or-nothing per SDK; it has to be all-or-nothing per call site.

Step 2: build the switch into the client

The split belongs in one place: the function that constructs your client. It needs a stable cohort per user, so a user does not bounce between paths mid-conversation, and a share you can change without a deploy. Fowler describes this as a canary cohort created by "a random sampling of 1% of the user base", consistently routed by user². Operational switches like this one need to change fast; waiting for a release to flip one defeats the purpose².

import hashlib
import os

from openai import OpenAI

_direct = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    timeout=60.0,
    max_retries=2,
)
_gateway = OpenAI(
    base_url="https://api.infro.io/v1",
    api_key=os.environ["INFRO_API_KEY"],
    timeout=60.0,
    max_retries=1,
)


def gateway_share() -> float:
    # Read from your flag service or config store, not from code.
    return float(os.environ.get("GATEWAY_SHARE", "0"))


def in_canary(user_id: str) -> bool:
    bucket = int(hashlib.sha256(user_id.encode()).hexdigest()[:8], 16) % 10_000
    return bucket < gateway_share() * 10_000


def client_for(user_id: str) -> tuple[OpenAI, str, str]:
    """Return the client, the model id for that path, and the cohort label."""
    if in_canary(user_id):
        return _gateway, "openai/gpt-5.6-sol", "canary"
    return _direct, "gpt-5.6-sol", "control"

client, model, cohort = client_for(user.id)
extra = {"metadata": {"feature": "support-reply"}} if cohort == "canary" else {}

response = client.chat.completions.create(
    model=model,
    messages=messages,
    stream=False,
    extra_body=extra,
)
metrics.increment("llm.requests", tags={"cohort": cohort})

The same shape in TypeScript, where the SDK takes baseURL:

import OpenAI from "openai";
import { createHash } from "node:crypto";

const direct = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, timeout: 60_000, maxRetries: 2 });
const gateway = new OpenAI({
  baseURL: "https://api.infro.io/v1",
  apiKey: process.env.INFRO_API_KEY,
  timeout: 60_000,
  maxRetries: 1,
});

export function clientFor(userId: string, share: number) {
  const bucket = parseInt(createHash("sha256").update(userId).digest("hex").slice(0, 8), 16) % 10_000;
  return bucket < share * 10_000
    ? { client: gateway, model: "openai/gpt-5.6-sol", cohort: "canary" as const }
    : { client: direct, model: "gpt-5.6-sol", cohort: "control" as const };
}

Three details in that code are deliberate. The timeout is explicit, because the Python SDK's default is ten minutes³, far longer than any user will wait. The gateway client retries less, because the gateway already moves a failed attempt to another connection before the first token. And the cohort label is emitted on every metric, because the whole method depends on comparing the two populations.

The SDK also reads OPENAI_BASE_URL from the environment³. That is convenient for a full cut-over and wrong for a canary: it moves every request in the process at once.

Step 3: compare populations, not days

The workbook is specific about evaluation: compare the canary against a control population running at the same time, because before-and-after comparisons pick up every other change in the system¹. It also shows why aggregate dashboards hide canary problems: a 5% canary with a 20% error rate adds only one point to the overall error rate¹. Break every signal down by cohort.

Two comparisons deserve care:

• Latency. The gateway adds a network hop. Decide before you start how much extra time to first token is acceptable, then measure it from the client in both cohorts rather than trusting either side's server timings alone.
• Output shape. The model is the same, so finish_reason distribution and tokens per request should match. A shift means something in the request changed on the way, such as a default max_tokens, and is worth finding before it reaches every user.

Step 4: advance in stages

Each stage should last long enough to include a daily peak, and only one change should be in flight at a time, since overlapping canaries make it impossible to say which one caused a signal¹.

Step 5: roll back without a deploy

Rollback is setting the share to zero. Because the cohort is computed per request from a configuration value, the next request from every user goes direct again. Nothing needs to be migrated back: there is no state on the gateway path that the direct path depends on.

Keep the rollback path real until the end:

1. Leave the direct client and OPENAI_API_KEY in place through a soak period at 100%.
2. Remove the direct call sites in one change, after the soak.
3. Then revoke the OpenAI key the application held, so the only copy of an OpenAI credential is the one sealed in the gateway connection.

Running the canary against INFRO

In INFRO the gateway path calls your own OpenAI account: you connect an OpenAI API key under Providers, it is sealed and never returned, and your provider bills you directly (connect a provider). INFRO checks the key when it is saved, so a typo shows up there rather than in the first canary request.

A few behaviours shape the runbook:

• Model ids. openai/gpt-5.6-sol resolves to every enabled, healthy connection that serves OpenAI models, in creation order (connect a provider). The response's model field carries that catalog id, so any code comparing response.model with a bare name needs updating.
• Extra fields. Responses add provider, connection, route and usage.cost, an estimate of what OpenAI will bill for the request, null when unpriced (chat completions). The top-level id equals the X-INFRO-Request-Id header, which is the id to log beside your own.
• Environments fail closed. A key that names no environment cannot reach a connection scoped to production. Mismatches return 503 connection_out_of_scope, not a provider error (connect a provider).
• Refusals that never reach OpenAI. Budget, subscription and allowlist refusals return 402 or 403 immediately without failover (fallbacks). They are policy working, and they should be expected in the canary before they surprise production.
• Shared limits. Both cohorts spend the same OpenAI rate limit. A provider 429 returns 429 provider_rate_limited with OpenAI's Retry-After, and it does not count against the connection's health (connect a provider).

To read the canary side, filter the request log by feature and outcome:

curl "https://api.infro.io/v1/requests?feature=support-reply&status=error&limit=50" \
  -H "Authorization: Bearer $INFRO_API_KEY"

Each record carries ttft_ms, finish_reason, token counts and every fallback attempt, and the key's history access decides which requests it can read (request traces).

After the migration, the next change is easier. A route with the weighted strategy picks the first target in proportion to its weight and still fails over to the rest, so a second provider can be canaried with a weight change instead of a code change. With the management API that is one versioned write:

curl https://api.infro.io/manage/v1/routes/rt_01J9Z6 \
  -X PATCH \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H 'If-Match: "4"' \
  -H "Content-Type: application/json" \
  -d '{
    "strategy": "weighted",
    "targets": [
      {"connection_id": "conn_openai", "model": "openai/gpt-5.6-sol", "weight": 95},
      {"connection_id": "conn_azure", "model": "openai/gpt-5.6-sol", "weight": 5}
    ]
  }'

targets replaces the whole list, and If-Match refuses the write if someone changed the route since you read it (management API resources).

Checklist

• Every call site sorted into gateway or direct, with the Responses API and stateful APIs kept direct.
• Client factory with a hashed, stable cohort and a share read from configuration.
• Explicit timeouts on both clients; fewer SDK retries on the gateway client.
• Every metric tagged with the cohort; every gateway request tagged with a feature.
• Gate thresholds written down before stage 1.
• Rollback tested once at stage 0 by setting the share to zero.
• Direct path kept through the soak; OpenAI key removed from the application afterwards.