A model API key is a spending instrument. Whoever holds it can run inference that someone else pays for, at whatever rate the account's quotas allow. Most advice about these keys stops at "keep them out of source control". This guide covers what comes after: how a team that runs AI traffic through a gateway should issue, scope, limit, rotate and pause the keys its applications hold, and what to do in the first hour after one leaks. Storage basics are in how to secure LLM API keys; this article assumes them.
Two kinds of key, and why the application holds only one
A gateway setup has two credentials per call. The provider key authenticates the gateway to OpenAI, Anthropic, Bedrock or whichever provider serves the request. The gateway key authenticates your application to the gateway. They should never be the same thing, and the application should only ever see the second.
The reason is blast radius. A provider key is usually broad: it can call every model the account can reach, at the account's full quota, from anywhere on the internet. Sysdig's threat research team named the abuse of stolen model credentials LLMjacking in May 2024: attackers who took cloud credentials from a server with an unpatched framework vulnerability tested them against ten AI services, checked whether prompt logging was on, and fronted the access with a reverse proxy so others could use it while the victim paid. Sysdig estimated the worst case for the account it observed at about $46,000 a day in model charges1.
A gateway key can be much narrower than that: one environment, a chosen set of providers, a list of models, a set of source networks, a spend ceiling and a rate. If it leaks, the attacker gets that, not the provider account. The provider key stays sealed in one place, held by one consumer. How a gateway should seal and scope those provider credentials is covered in BYOK credential architecture, and how to narrow the provider-side keys themselves in least privilege for provider admin and usage keys.
holds
Your applicationOne gateway key per service and environment, read from a secrets manager. Never a provider key.carries
Gateway keyEnvironment, project, providers it may use, allowed networks, model list, spend limit, rate tier, expiry, on/off.decides
The gatewayRefuses, narrows or admits each request, and records which key sent it.sealed
Provider credentialsYour own provider keys, held server-side and never returned to an application.
One key per service, per environment
The default that causes the most trouble is one key shared by everything. It makes spend impossible to attribute, turns every rotation into a coordinated deploy across services, and means revoking a leaked key takes every service down with it.
Issue one key per service and per environment instead. OpenAI's production guidance recommends separate projects for development and production so testing cannot disturb the live application, restricting who can reach the production project, and setting rate and spend limits per project2. OWASP's secrets guidance says the same about secrets in general: different per environment, and ideally kept apart from development secrets entirely3.
Environment scoping is only useful if it fails closed. A key that says nothing about its environment should not reach a provider connection someone deliberately reserved for production. Otherwise the easiest key to create, the one where nobody thought about environments, is also the most powerful.
| Can the key reach the connection? | Connection: all environments | Connection: production | Connection: staging + development |
|---|---|---|---|
| Key: production | ●Yes | ●Yes | ○No |
| Key: staging | ●Yes | ○No | ●Yes |
| Key: no environment | ●Yes | ○No | ○No |
Limit what a key can reach
Providers and models. A summarization service has no business calling an image model, and a service approved for one vendor's terms should not quietly fail over to another's. Limit each key to the providers it needs, and to the models it needs, and make sure the limit holds on every path: a direct model name, a named route, and a fallback tried after a failure. The governance side of model lists is in model access governance.
Networks. An IP allowlist turns a stolen key into a useless string unless the attacker is also inside your network. It works well for servers with stable egress addresses (a NAT gateway, fixed egress IPs, a VPC with a known range) and badly for anything whose addresses change, such as some serverless platforms or developer laptops. Two details matter. The gateway must use the client address it observes at its edge, not a header the client can set, such as X-Forwarded-For. And when no trusted address is available, a key with an allowlist should be refused, not let through.
Cap what a key can spend, and how fast
OWASP's unbounded consumption risk names denial of wallet, running up cost against a pay-per-use AI service, and lists rate limiting and user quotas among the mitigations4. On a key, that means two independent limits:
- A spend limit bounds money. A resetting limit (daily, weekly, monthly) catches a runaway loop within one window. A lifetime limit that never resets is the strongest leak control there is: it bounds the total a key can ever cost, however long it goes unnoticed.
- A rate limit bounds speed. It cannot cap total cost on its own, but it slows abuse enough for an alert to reach someone, and it stops one misbehaving service from consuming the provider quota other services need.
OpenAI also supports hard spend limits on its side2, and those are worth setting too: a gateway's spend figure is an estimate of what the provider will bill, not the provider's own meter.
Rotate with an overlap window
Rotation is where most teams cause their own outages. Revoke-then-replace breaks every service still holding the old key. The pattern that avoids it is the one OpenAI describes for its own keys — create the replacement, update applications, revoke the old key once the new one is confirmed working2 — with a defined overlap during which both keys work. OWASP's cheat sheet describes the same sequence as create, set, test, finish3.
- TRotateA new secret, shown once, with every setting of the old key. The old key keeps working.
- T + minutesDeploy the new secretUpdate the secrets manager entry the service reads, then roll the service.
- T + verifyConfirm traffic movedGET /v1/key from the deployed service answers with the new key.
- T + overlapOld key stops1 hour, 24 hours (default) or 7 days later it returns 401. A leaked key rotates with no overlap.
Three rules make rotation safe. The new key must inherit every setting of the old one, or rotation quietly widens or narrows access. Spend limits must carry across, or rotation becomes a way to reset a cap. And a leaked key rotates with no overlap at all.
An off switch that is not revocation
Revocation is final. That is right for a leaked key and wrong for most other reasons to stop traffic: a misbehaving deploy, a customer dispute, an investigation in progress, a load test that got out of hand. For those, a key needs an off switch that refuses every request but keeps the secret, the scope and the limits, so turning it back on restores the same key without redeploying anything.
The off switch is also the fastest containment step when you suspect a leak but have not confirmed it: it stops spend immediately and can be undone if the alarm was false. Once a leak is confirmed, follow it with rotation or revocation, because a disabled secret that someone later re-enables is still the leaked secret.
People who manage keys
Keys are created by people, and scoping people matters as much as scoping keys. A developer working on one product should be able to create and rotate that product's keys without being able to read other projects' traffic, change organization-wide budgets or edit provider connections. Every key change (created, rotated, disabled, revoked) belongs in an append-only audit log with the actor and time, so an investigation can tell who issued the key that leaked and what else they changed.
When a key leaks
Leaks are usually found in one of three ways: a secret scanner, a spend alert, or a provider notice. GitHub runs secret scanning automatically and for free on public repositories, across the whole Git history, and for partner-validated secrets it notifies the issuing provider, which can revoke the credential5. That means a provider key pushed to a public repository may be revoked by the provider before you hear about it — another reason applications should not hold provider keys.
OWASP's remediation order for an exposed secret is revoke, rotate, delete the secret from systems, code and logs, and keep the access and rotation history for responders3. Applied to AI keys:
first
Contain (minutes)
- Turn the key off, or rotate with no overlap
- If it was a provider key, revoke it at the provider
- Check what else lived beside it: cloud credentials, .env files
then
Scope (hours)
- Pull the key's requests: models, volume, source networks
- Read the audit log for changes made with the same access
- Compare estimated spend with the provider's own usage data
after
Harden (days)
- Add an IP allowlist and a lifetime spend cap
- Narrow the key's providers and models
- Find how it leaked and remove it from history and logs
Scoping the damage is where per-key records pay off. With one key per service you can pull exactly that key's requests (which models, how many, from which networks), compare the gateway's estimate with the provider's own usage data, and read the audit log for anything else changed with the same access. Sysdig observed attackers checking whether prompt logging was enabled before using stolen credentials1, so look for unusual reads and configuration changes as well as inference. The broader incident workflow is in LLM incident response.
Gateway keys in INFRO: scope, limits, rotation and the off switch
INFRO is a BYOK control plane: customers connect their own provider accounts, and applications call the gateway with an INFRO key. Provider credentials stay sealed in INFRO and are never returned to an application.
A key's scope. Keys start with sk_infro_ and are created in the console under Keys. Each carries a label, an optional spend limit, a rate-limit tier, and optionally an environment, a project, an expiry, a model allowlist, a route allowlist, the providers it may use and its allowed networks. GET /v1/key returns the label, limit, usage, remaining room and next reset of whichever key made the call, which is how a deployed service confirms it picked up the right key.
Environments. Every organization starts with production, staging and development and can add its own. A provider connection or route serves one environment, several or all of them; a key names one environment or none. Matching is fail-closed in both directions: a key reaches connections that list its environment and connections set to every environment, and a key with no environment reaches only the latter.
Providers. By default a key may use every connected provider, including ones connected later. Only selected (or allowed_providers on POST / PATCH /console/keys and the management API) narrows it to provider ids such as openai. The limit holds on a model name, a provider's own model id, a route and per-request fallbacks; a route target on another provider is skipped. When nothing the key may use can serve a request, the gateway answers 503 with code connection_out_of_scope, naming the providers the key allows. A rotated key keeps the same providers.
Allowed networks. allowed_ips takes up to 32 IPv4 or IPv6 addresses or CIDR ranges per key. The check uses the client address the Cloudflare edge reports, never X-Forwarded-For, and fails closed when no trusted address is available. A refused request returns 403 with code ip_not_allowed before any budget, routing or provider call, and the response never lists the allowed networks. It applies to every request a gateway key authenticates (inference, /v1/usage, /v1/requests, /v1/key and the FOCUS export), not to management tokens or console sign-in.
Spend and rate limits. A key's spend limit is a ceiling on estimated provider cost that resets daily (midnight UTC), weekly (Monday, midnight UTC), monthly (on the organization's budget cycle day) or never (a lifetime cap). At the limit, requests return 402 budget_exceeded with code key_budget, or key_lifetime_limit for a lifetime cap, while other keys keep working. Under a blocking limit, a model with no known price is refused (unpriced_model_under_budget) by default, because its cost cannot be counted. Rate limits are per key, from fixed tiers of 60 to 50,000 requests per minute (200 by default), independent of the plan; over the limit, the gateway returns 429 rate_limit_exceeded with Retry-After.
Rotation. Rotate… in the console, POST /console/keys/{id}/rotate, or POST /manage/v1/gateway_keys/{id}/rotate issues a new secret, shown once, with every setting of the old key. The console offers an overlap of none (the old key is revoked at once), 1 hour, 24 hours (the default) or 7 days; the API's overlap_seconds takes any whole number of seconds up to 7 days. The old key never outlives its own expiry. Both keys share one spend limit and one lifetime cap, so rotating never creates fresh spending room, and analytics show the old key's history on its replacement.
The off switch. A key can be turned off from the console or with PATCH and "disabled": true. Requests with it return 403 with code key_disabled; its secret, scope and limits are kept, and turning it back on makes the same secret work on the next request. Whoever may revoke a key may turn it off. Revocation itself is immediate and cannot be undone: later requests return 401.
| INFRO gateway answers | Status and code | Retry? |
|---|---|---|
| Revoked, expired or unknown key | 401 authentication_error | ○No |
| Key turned off | 403 key_disabled | ○, turn it on |
| Outside allowed networks | 403 ip_not_allowed | ○No |
| Model not allowed | 403 model_not_allowed | ○No |
| Key spend limit reached | 402 key_budget or key_lifetime_limit | ○, raise it or wait for reset |
| Over the key's rate tier | 429 rate_limit_exceeded | ●, after Retry-After |
| No allowed provider can serve | 503 connection_out_of_scope | ○No |
People. An owner or admin can limit a developer or analyst to some projects. A member limited this way manages the gateway keys in those projects, and every key they create or move belongs to one of them; budgets, provider and routing settings, policies, members and the audit log are closed to them, and the gateway enforces this on every request.
Audit. The audit log records api_key.created, api_key.updated, api_key.rotated, api_key.disabled, api_key.enabled and api_key.revoked, plus member.project_access_changed and environment changes, each with the actor and time.
Plans. Environments, provider limits, allowed networks, rotation, per-key spend and rate limits and the audit log are on every plan. The authentication reference and rate and spend limits have the full request and response shapes.
A key hygiene checklist
- Applications hold gateway keys only; provider keys stay sealed server-side.
- One key per service and per environment, each with a label someone recognizes.
- Each key limited to the providers and models it needs.
- Production keys restricted to your servers' egress networks.
- A resetting spend limit and a lifetime cap on every key, plus a rate tier sized to the service.
- Rotation rehearsed with an overlap window; leak rotation with none.
- Someone knows where the off switch is, and the audit log is reviewed after every key change.