BYOK gateway architecture: sealing, scoping and rotating keys

A gateway that calls AI providers on a customer's behalf is, before it is anything else, a place where credentials that spend money live. Under a bring-your-own-key model those credentials belong to accounts the gateway operator does not own: the customer's OpenAI organization, their Anthropic workspace, their Azure resource. Losing one is not an inconvenience for the operator. It is a bill, and possibly a data exposure, on somebody else's account.

That changes the design questions. Where is the credential encrypted, and under what key? What stops a sealed value being moved to another tenant? Who can ever read it back? And when something leaks, which of several very different rotations is the right one?

Name the credentials first

A BYOK gateway usually handles four kinds of credential, and they fail differently. Treating them as one "API keys" problem is how a design ends up encrypting what it should hash and showing what it should never return.

The split that matters is direction. Inbound credentials — the keys your services send to the gateway, and the tokens your automation uses to configure it — only ever need to be verified. Outbound credentials — the provider keys the gateway presents on your behalf — must be recovered in plaintext at request time. Verification allows a one-way hash. Recovery requires encryption, which means a key that can decrypt, which means the whole design turns on where that key lives.

Provider documentation frames the stakes the same way. OpenAI's API reference tells developers an API key is a secret not to be shared or exposed in client-side code¹. OWASP's 2025 list for LLM applications includes sensitive information disclosure and unbounded consumption as separate risks²; a leaked provider credential is both at once.

Sealing outbound credentials

Use an authenticated encryption mode. AES in Galois/Counter Mode is specified by NIST as authenticated encryption with associated data³: it encrypts the value and produces a tag that fails verification if the ciphertext, the nonce or the associated data changed. That last input is the one most implementations leave empty, and it is the one that stops a class of attack specific to multi-tenant storage.

Without associated data, anyone who can write to the credentials table — a SQL injection, a mistaken support script, a compromised migration — can copy the sealed blob from tenant A's row into tenant B's. The blob decrypts perfectly, and tenant B's traffic now runs on tenant A's account. With a binding label such as conn:<organization>:<connection> supplied as associated data, the blob only opens in the row it was sealed for. Moved anywhere else, the tag check fails and the request fails closed.

const IV_BYTES = 12; // 96-bit nonce, random per seal, never reused with one key

function label(org: string, connection: string): Uint8Array {
  return new TextEncoder().encode(`provider-credential:conn:${org}:${connection}`);
}

export async function seal(plaintext: string, kek: CryptoKey, org: string, conn: string) {
  const iv = crypto.getRandomValues(new Uint8Array(IV_BYTES));
  const ct = new Uint8Array(
    await crypto.subtle.encrypt(
      { name: "AES-GCM", iv, additionalData: label(org, conn) },
      kek,
      new TextEncoder().encode(plaintext),
    ),
  );
  const packed = new Uint8Array(iv.length + ct.length); // one column: iv ‖ ciphertext ‖ tag
  packed.set(iv, 0);
  packed.set(ct, iv.length);
  return { secret: btoa(String.fromCharCode(...packed)), alg: "aes-256-gcm.v1", last4: plaintext.slice(-4) };
}

export async function open(secret: string, kek: CryptoKey, org: string, conn: string) {
  const packed = Uint8Array.from(atob(secret), (c) => c.charCodeAt(0));
  const iv = packed.subarray(0, IV_BYTES);
  // Throws if the blob was moved to another row: the label no longer matches.
  const pt = await crypto.subtle.decrypt(
    { name: "AES-GCM", iv, additionalData: label(org, conn) },
    kek,
    packed.subarray(IV_BYTES),
  );
  return new TextDecoder().decode(pt);
}

Three smaller decisions are worth copying. Pack the nonce and ciphertext into one column so a partial write cannot leave them disagreeing. Store an algorithm version beside the blob so a future scheme can coexist with this one instead of forcing a flag day. And keep the key-encryption key out of the database entirely — in a platform secret or a KMS — so a database dump alone opens nothing.

Never return what you sealed

A "reveal" button is the feature that turns one compromised console session into every provider credential an organization holds. Make outbound secrets write-only: accepted on create and on an explicit replace call, never included in a read, an export, an infrastructure-as-code state file or a support tool. Show the last four characters and when it was set; that is enough to tell two keys apart and to confirm a rotation landed.

The cost is real and should be stated to users: a gateway that cannot show a credential cannot recover a lost one either. The recovery path is to issue a new credential at the provider and replace it.

Hash inbound keys instead of encrypting them

A gateway's own keys are verified, not recovered, so there is no reason to keep anything that could be turned back into them. Generate them from a cryptographic random source with enough entropy that guessing is not a strategy, and store a digest. A slow password hash buys nothing here: it protects low-entropy human secrets from dictionary attacks, and 32 random bytes have no dictionary, while a deliberately slow function adds its cost to every request. Keep a short display prefix and the last four characters so humans can recognise a key in a list.

Revocation deserves one detail. If key lookups are cached at the edge, write revocation as a tombstone rather than deleting the row, so a stale cache entry cannot resurrect a key that was just killed.

Scope so that defaults fail closed

Scoping decides what a stolen key can reach, and the most important scoping rule is about the key nobody thought about. When a provider connection can be marked production and a gateway key can name an environment, the matching must narrow in both directions: a key that names no environment should reach only unscoped connections, not everything.

Layer the other scopes on the same principle — each non-empty list narrows, none widens:

• a project per application, so history and budgets separate cleanly;
• a model allowlist per organization, project and key, all of which must pass;
• a network allowlist per key, checked against the address the edge observed rather than a header the client controls;
• a spend limit per key, so a leak is a bounded loss;
• a separate history permission, so a key that can call models cannot automatically read every other application's requests.

Keep configuration authority on a different credential from traffic authority. A key that serves requests should be refused by the configuration API, and a configuration token should be refused by the inference API. Then a leaked application key cannot rewrite the policy that limits it.

Three rotations, not one

"Rotate the key" means three different operations in a BYOK gateway, and conflating them is how a routine task becomes an outage.

Rotating a gateway key is the frequent one. The safe shape issues a new secret that inherits every setting of the old key, keeps both valid for an overlap window while the new one is deployed, and lets the old one expire. OpenAI, for its own keys, recommends setting an expiration when a key is created and establishing a regular rotation process⁴, and notes that revocations take effect within a few seconds¹. After a confirmed leak the overlap should be zero.

Replacing a provider credential happens when the customer rotates at their provider. The gateway swaps the sealed value and uses it from the next request, but it cannot revoke the old key at the provider — only the account owner can. Say so in the interface, or the old key lives on at the provider indefinitely.

Rotating the key-encryption key is rare and is an operator procedure. NIST defines a cryptoperiod as the time a key is authorized for use and lists, among its purposes, limiting the exposure if a single key is compromised; a compromised key's cryptoperiod is no longer valid⁵. Re-sealing every stored credential under a new key should be one transaction that verifies each new blob decrypts before writing it, preserves each row's binding label exactly, and can be re-run safely.

How INFRO seals and scopes credentials

These are the mechanisms in INFRO's gateway today, as documented in Connect a provider, Authentication and Management tokens:

• Provider credentials are encrypted with AES-256-GCM under a key held as a Worker secret, not in the database, with the organization and connection bound in as additional authenticated data. A credential connected for cost reconciliation is sealed separately under its own binding.
• No endpoint, console screen, export or support tool returns a provider credential. The console shows the last four characters and when it was set. Replacing one takes effect on the next request and does not revoke the old key at the provider.
• Gateway keys start with sk_infro_, are stored as a SHA-256 digest with a display prefix and the last four characters, and can carry a label, project, environment, expiry, spend limit, model allowlist and up to 32 allowed networks. A request from outside them returns 403 ip_not_allowed before any budget check or provider call.
• Rotating a gateway key copies every setting to the new key and offers an overlap of none, 1 hour, 24 hours (the default) or 7 days; old and new share one spend limit, so rotation never creates fresh spending room. It is recorded as api_key.rotated in the audit log.
• Management tokens (mt_infro_) are refused on /v1, gateway keys are refused on /manage/v1, and every management token expires, at most 365 days after creation.

The same controls through the management API, replacing a provider credential after rotating it at the provider:

curl -X POST https://api.infro.io/manage/v1/provider_connections/$CONNECTION_ID/credential \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"secret": "'"$NEW_PROVIDER_KEY"'", "validate": true}'

A review checklist

• Every credential type is named, with who holds it and what it can do.
• Outbound secrets use authenticated encryption with a per-row binding label.
• The key-encryption key is outside the database, and its loss is covered by a written recovery procedure.
• No read path returns an outbound secret; only the last four characters are shown.
• Inbound keys are random, hashed at rest, and revoked with a tombstone.
• A key with no environment reaches only unscoped connections.
• Configuration and traffic use different credentials, each refused by the other's API.
• Gateway-key rotation supports an overlap and a zero-overlap revocation.
• Replacing a provider credential prompts the user to revoke the old one at the provider.