What is a BYOK AI gateway (and when you don't need one)

What BYOK changes

"Bring your own key" decides who the provider's customer is. Without it, the gateway buys inference on its own accounts and resells it to you. With it, the request runs on your account, and the gateway only governs it.

For a prototype, the left column is simpler: one account, one balance, every model. For a company with a contract, committed spend with a cloud, or a data processing agreement, the right column is usually required.

What the gateway does with your keys

Underneath, a BYOK gateway does what this sample does across two OpenAI projects you own, and then much more: it moves a request to your next key when one is rate limited or failing, and refuses to move it when a spend limit said stop.

import OpenAI, { APIError } from "openai";

// Two keys from two OpenAI projects you own: the same contract and rates,
// separate rate limits. Every request stays on your own account.
const clients = [process.env.OPENAI_API_KEY, process.env.OPENAI_API_KEY_STANDBY].map(
  (apiKey) => new OpenAI({ apiKey, maxRetries: 0, timeout: 20_000 }),
);

const SPEND_CODES = new Set([
  "credit_balance_exhausted",
  "organization_spend_limit_exceeded",
  "project_spend_limit_exceeded",
  "organization_usage_limit_exceeded",
  "insufficient_quota",
]);

// Move to the next key on a rate limit or a provider failure. Never on a spend
// limit: the next project would spend around the limit you set on this one.
function tryNextKey(err: unknown): boolean {
  if (!(err instanceof APIError)) return false;
  if (err.status === 429) return !SPEND_CODES.has(String(err.code)) && !SPEND_CODES.has(String(err.type));
  return err.status === undefined || err.status >= 500;
}

export async function complete(prompt: string): Promise<string> {
  let lastError: unknown;
  for (const [i, client] of clients.entries()) {
    try {
      const res = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [{ role: "user", content: prompt }],
      });
      return `[key ${i + 1}] ${res.choices[0]?.message.content ?? ""}`;
    } catch (err) {
      if (!tryNextKey(err)) throw err;
      lastError = err;
    }
  }
  throw lastError;
}

try {
  console.log(await complete("Draft a reply to this refund request."));
} catch (err) {
  if (!(err instanceof APIError)) throw err;
  console.error(`stopped: ${err.status} ${err.code ?? err.type}`);
  process.exitCode = 1;
}

import os
import sys

import openai

# Two keys from two OpenAI projects you own: the same contract and rates,
# separate rate limits. Every request stays on your own account.
clients = [
    openai.OpenAI(api_key=key, max_retries=0, timeout=20.0)
    for key in (os.environ["OPENAI_API_KEY"], os.environ["OPENAI_API_KEY_STANDBY"])
]

SPEND_CODES = {
    "credit_balance_exhausted",
    "organization_spend_limit_exceeded",
    "project_spend_limit_exceeded",
    "organization_usage_limit_exceeded",
    "insufficient_quota",
}


def try_next_key(err: Exception) -> bool:
    """Move on after a rate limit or a provider failure. Never on a spend limit:
    the next project would spend around the limit you set on this one."""
    if isinstance(err, openai.APIConnectionError):
        return True
    if not isinstance(err, openai.APIStatusError):
        return False
    if err.status_code == 429:
        return err.code not in SPEND_CODES and err.type not in SPEND_CODES
    return err.status_code >= 500


def complete(prompt: str) -> str:
    last_error: Exception | None = None
    for i, client in enumerate(clients, start=1):
        try:
            res = client.chat.completions.create(
                model="gpt-5.4-mini",
                messages=[{"role": "user", "content": prompt}],
            )
            return f"[key {i}] {res.choices[0].message.content}"
        except Exception as err:
            if not try_next_key(err):
                raise
            last_error = err
    raise last_error


if __name__ == "__main__":
    try:
        print(complete("Draft a reply to this refund request."))
    except openai.APIStatusError as err:
        print(f"stopped: {err.status_code} {err.code or err.type}", file=sys.stderr)
        sys.exit(1)

A gateway adds the rest: the same across providers, budgets that refuse a request before the provider is called, a cost estimate on every request, keys that your applications never see, and a record of who changed what. That is the product. BYOK is only how the requests are paid for.

When you do not need one

• One provider, one service. OpenAI already lets you set hard limits per organization and per project. If that covers you, a gateway adds a hop and little else.
• You want models without accounts. If you would rather not hold an account with each provider, BYOK is the wrong model: an aggregator that buys inference for you fits better, and you accept its fee and its terms.
• You must run everything yourself. A self-hosted gateway such as LiteLLM (MIT licensed) keeps keys and prompts in your network, at the cost of running a server and its Postgres database, and being on call for it.

You need one when two or more of these are true: more than one provider or account, several teams or services spending, a need to cap spend per customer or per key, or a need for failover you do not want to build.

Five questions that separate BYOK gateways

1. What happens when my key fails? This matters most, because a fallback onto the gateway's own capacity runs a request you meant to keep on your contract on theirs.

2. Do budgets see BYOK spend? A limit that cannot see your own-key traffic does not limit it. Vercel's budgets do not count BYOK spend. OpenRouter's count it only when "Include BYOK spend" is on. Cloudflare's spend limits count BYOK requests for models with known pricing.

3. What does BYOK cost?

4. How are keys stored, scoped and rotated? Ask whether a key is encrypted at rest and never shown again, whether you can hold several keys per provider and rotate one while the others serve, and whether every key change is in an audit log. On your side, give the gateway an inference-only key, never an admin key.

5. How do I leave? With your own accounts, leaving should mean pointing your applications back at the provider. Check that the API is a standard shape and that your request and cost history can be exported.

Setting up BYOK safely

1. One key per gateway and environment. Never reuse the key your own services use, so revoking the gateway's access breaks nothing else.
2. Scope it at the provider. An OpenAI project key for the project the gateway serves; an Anthropic key in a dedicated workspace.
3. Keep a provider-side ceiling. Set OpenAI's project hard limit (or Anthropic's workspace limit) as the backstop if the gateway's own limits are misconfigured. How to set it.
4. Record the arrangement and rehearse revoking and replacing the key once before you depend on it.

Do you need a BYOK gateway?

No gateway

if one service calls one provider, and the provider's project limits and dashboard are enough.

An aggregator

if you want every model behind one balance and do not want provider accounts; you accept its fee and its fallback rules.

A self-hosted gateway

if keys and prompts must stay in your network and you can run and patch a service in the request path.

A hosted BYOK gateway

if you hold provider contracts and need failover, budgets and cost per customer across them without running it yourself.

Questions

What does BYOK mean in AI?

Bring your own key: a platform runs your AI requests on provider accounts you own, with credentials you give it, instead of on its own accounts. Your provider bills you directly at your own rates.

Does a BYOK gateway keep my negotiated rates?

Yes, as long as every request runs on your account. Check what the gateway does when your key fails: some fall back to their own capacity, which is billed at their price, not your contract's.

Does a BYOK gateway see my prompts?

It is in the request path, so it handles them. Ask whether it stores them. INFRO stores prompts and responses only when an owner or admin turns capture on.