Zero trust for AI API access: a reference architecture

What does zero trust mean for AI API access? No caller is trusted because of where it runs or which shared key it holds. Every service, agent and developer that calls a model authenticates with its own credential; every request is checked against policy — which models, which providers, which environment, which networks, how much spend — before the provider is called; decisions and usage are logged; and provider credentials stay in one enforcement point rather than in applications. Provider-side controls cover the traffic that enforcement point cannot see.

The common starting point is the opposite: one provider key in an environment variable, shared by every service and several laptops, able to call every model the account can, with no decision made per request. This guide maps the zero trust tenets of NIST SP 800-207 onto AI API calls, lays out a reference architecture in a table, and names its limits.

What NIST SP 800-207 says, briefly

NIST SP 800-207, Zero Trust Architecture, was published in August 2020. Its abstract describes zero trust as shifting defenses from network perimeters to users, assets and resources, and states that no implicit trust is granted to assets or user accounts based solely on their network location or ownership¹. Section 2.1 sets out seven tenets; section 3.1 defines the logical components that enforce them, chiefly a policy engine that decides, a policy administrator that executes the decision, and a policy enforcement point that enables, monitors and terminates connections between a subject and a resource².

CISA's Zero Trust Maturity Model builds on the same ideas with maturity stages from traditional through initial and advanced to optimal³. Neither document mentions AI APIs. The mapping below is an interpretation, and it is ours.

The seven tenets, applied to AI API calls

The tenets' own text supports two details. Tenet 3 says access should be granted with the least privileges needed and that authorization to one resource does not grant access to another. Tenet 4 lists observed behaviour and measured deviations from usage patterns among the attributes a policy may use² — which, for AI calls, is where spend comes in.

A reference architecture

This is the architecture in components, the zero trust role each plays, and what it does for an AI call.

The flow for one request, in order:

1. The caller sends a request with its own credential to the gateway.
2. The gateway authenticates the credential and resolves its project, environment and owner.
3. Network conditions are checked: a request from outside the credential's allowed ranges is refused.
4. Policy is evaluated: is this model allowed for this organization, project and key; is this provider allowed; is this environment's connection in scope; does the budget at every level have room; do content rules pass on the prompt.
5. If any check fails, the request is refused with a specific error and the provider is never called.
6. If all pass, the gateway uses a provider credential from its vault to forward the request, applies content rules to the response, and records the outcome and estimated cost.
7. Telemetry compares usage with the usual pattern and raises an alert when it departs from it.

The five axes of least privilege

For AI calls, least privilege is not one setting. It has five axes, and a credential should be narrow on each.

• Models. An allowlist per organization, project and key, all of which must pass. A support bot does not need an image model or the most expensive reasoning tier. Model access governance covers the policy side.
• Providers. A caller that should use only one provider's models should not be able to reach another's, even when the model name is ambiguous.
• Environments. A development credential must never reach a production provider connection. The default must narrow: a credential that names no environment should reach less, not everything. Gateway key controls and environments explains why.
• Networks. An IP allowlist on a server-side credential means a copy used from elsewhere is refused. Under tenet 2 it is a condition added to identity, never a substitute for it.
• Spend. A hard cap per credential, project and organization, checked before the provider is called. OWASP's entry on unbounded consumption recommends rate limiting and user quotas per source, resource allocation management, and logging and monitoring to detect unusual patterns⁴.

Providers offer some of these axes on their own keys, and they belong in the design. OpenAI recommends setting an expiration date on project keys and supports organization-level limits on key lifetime⁵. Anthropic keys are tied to the workspace they were created in, and a workspace can carry its own spend limit below the organization's⁶. Google recommends API restrictions to reduce the impact of a compromised key, and short-lived service account credentials over API keys where possible⁷.

Continuous evaluation

Tenet 6 describes a constant cycle of obtaining access, assessing threats, adapting and re-evaluating trust². For AI calls that cycle has four parts:

• Per-request checks. Budgets and allowlists evaluated on every call, not at key issue time. A key that was fine yesterday is refused today when its project's budget is spent.
• Content checks. Pattern or classifier checks on prompts and responses — for credentials, card numbers, personal data — that block, redact or flag. Detection can miss things; treat it as one layer. LLM guardrails at the gateway goes deeper.
• Behavioural alerts. Spend or volume that departs from a scope's usual day, or a forecast that crosses a limit, raised to a person who can act. LLM spend anomalies explains the signals.
• Revocation that takes effect at once. OpenAI says a revoked key stops working within a few seconds⁸. A gateway credential should do the same, including at any cache in front of it.

Logging what the policy decided

Tenet 7 asks the enterprise to collect information and use it to improve policy. For AI calls, log at least the caller, project, environment, model, provider, decision, error code, token counts and estimated cost for every request, including refused ones. Whether to record prompt and response content is a separate decision with its own risks; an LLM prompt logging policy helps make it. Keep configuration changes — who changed an allowlist, a budget, a route — in an audit log that the people being audited cannot edit.

Where the architecture stops

An enforcement point governs the traffic that passes through it. Three gaps remain, and naming them is part of the design.

• Provider keys outside the vault. If a provider key also exists in an application's environment, on a laptop, or in a third-party tool, traffic on it never meets your policy. Keep provider keys in one place, and compare provider usage by key with what the gateway served to find traffic that bypassed it; detecting gateway bypass shows how.
• A stolen provider key. A key taken from the vault, or from anywhere else, can be used directly at the provider. Only provider-side scope, spend limits and revocation apply. What happens when a BYOK platform is breached models this.
• The enforcement point itself. It is now a high-value system. It needs strong sign-in, roles, an audit log and careful release practices of its own.

How INFRO maps to this architecture

INFRO is a gateway that can play the policy engine and enforcement point role for calls to the provider accounts you connect. What it enforces today:

• Identity per caller. INFRO gateway keys (sk_infro_, stored as a SHA-256 digest), each in one project and exactly one environment, never reaching a connection outside it. People sign in with two-factor authentication or passkeys, or with SAML or OIDC SSO and SCIM on the Business plan (SSO); roles and project access limit what each member can see and change.
• Least privilege. Model allowlists at organization, project and key level (403 model_not_allowed); keys limited to chosen providers; IP allowlists that refuse a request from outside them with 403 ip_not_allowed before any provider call; expiry and rotation with an overlap of none, 1 hour, 24 hours or 7 days.
• Continuous evaluation. Hard budgets per organization, project, member and key, checked before the provider is called (402); guardrails on prompts and responses on the Business plan; spend alerts when a scope's day departs from its usual day or a forecast crosses a limit, by email, Slack, Teams, PagerDuty or webhook.
• Logging. Every request in the request log with its decision and estimated cost (traces), prompt and response recording off unless you turn it on, and configuration changes in the audit log.
• The vault. Provider credentials encrypted with AES-256-GCM under a key held outside the database, never returned after saving.

The same limits apply to INFRO as to any enforcement point: its controls govern traffic that goes through it. A provider key used directly at the provider, including one stolen from INFRO, is limited only by the provider's own scope, spend limits and revocation, so set those too.

Summary

Zero trust for AI API access replaces one shared provider key with an identity per caller, a policy decision on every request, and a log of each decision. Put a single enforcement point between callers and providers, keep provider credentials there, narrow every credential on models, providers, environments, networks and spend, and evaluate on every call. Then cover what the enforcement point cannot see with provider-side scope, spend limits and usage reports.