What is an AI control plane? It is the layer between a company's applications and its AI providers that decides how requests run. It holds the provider credentials, routes and fails over requests, enforces which models each team may use and what each may spend before a provider is called, attributes cost and usage to projects and customers, and records who changed which policy. The providers that actually run the models are the data plane; the control plane is everything that decides, limits and records around them.
The term is borrowed from infrastructure, and the borrowing is precise. This guide explains what belongs in an AI control plane, how it differs from a gateway, an observability tool and an API gateway, and when a company needs one.
Where the term comes from
In networking, the data plane forwards packets and the control plane decides where they should go. Kubernetes uses the same split: its control plane components "manage the overall state of the cluster", while node components run the workloads1. The control plane holds intent — what should happen — and the data plane does the work.
AI traffic now has the same shape. The work is done by model providers: OpenAI, Anthropic, Google, a cloud platform, a self-hosted model server. The intent — which provider, which model, which key, how much spend, whose request — has to live somewhere. In most companies it starts scattered across application code, environment variables and spreadsheets. An AI control plane puts it in one place.
What an AI control plane does
| Function | What it decides or records | Without it |
|---|---|---|
| Credentials | Which provider accounts exist, sealed; which keys applications use | Provider keys copied into services and laptops |
| Routing and reliability | Which provider and model serve a request; failover order; retries | Hard-coded model names; outages handled in each service |
| Access policy | Which models a project, team or key may call | Anyone with a key can call any model |
| Spend control | Estimated cost of each request; budgets that refuse before the call | Overspend discovered on the invoice |
| Attribution | Cost and usage by project, key, feature and end customer | One provider bill nobody can split |
| Audit | Who created keys, changed routes, raised budgets | No answer for the security review |
| Lifecycle | Which models are being retired and what still calls them | Shutdown dates found in production errors |
Anthropic's own documentation describes much of this as what a gateway gives an organization: provider keys kept server-side with developers holding gateway credentials, usage attributed per developer or team, budgets and rate limits enforced in one place, audit logging, and provider switching without touching developer machines2. The control plane is that list treated as a product, across every provider and application rather than one tool.
Before the call, not after
The defining property is timing. A control plane acts before the provider is called: it can refuse a request from a key that may not use a model, or one that would exceed a budget, so the cost and the risk never occur. Recording what happened afterwards is necessary, but it is observability, not control. Several products in this market now enforce spend before the call; Cloudflare's AI Gateway, for example, blocks requests with a 429 once a spend limit is reached, and is explicit that its cost tracking is a best-effort estimate3. Every control plane works from estimates; the useful ones say so and check them against the provider's bill.
Control plane, gateway, observability, API gateway
| Layer | In the request path? | Understands models and tokens? | Decides before the call? | Typical scope |
|---|---|---|---|---|
| API gateway | Yes | No | Auth, rate limits | All HTTP APIs |
| LLM gateway | Yes | Yes | Routing, retries, sometimes limits | AI requests |
| Observability tool | No (traces sent to it) | Yes | No | Records and evaluation |
| AI control plane | Yes, through its gateway | Yes | Access, budgets, routing | Policy, cost and audit across an organization |
The lines blur because products combine layers. The useful question about any product is not what it calls itself but which of the functions in the first table it performs, and whether it performs them before or after the provider call. What is an LLM gateway covers the request-path part in depth.
Whose accounts does it run on?
The most consequential design choice in an AI control plane is commercial, not technical: whose provider accounts carry the traffic.
- The platform's accounts. The platform buys model access and charges you for it. You get one supplier relationship, and you lose your own contracts, negotiated rates, data agreements and the ability to leave without changing who runs your inference.
- Your accounts (BYOK). You connect provider accounts you already hold. Your providers bill you directly at your own rates; the control plane governs the traffic without sitting in the money flow.
Many products offer both, and the details matter: what happens when your own key fails (does the request fall back to the platform's capacity?) and whether budgets count spend on your own keys. What is BYOK for AI explains the model and the questions to ask.
When a company needs one
Signs that the layer is overdue:
- More than one team calls AI providers, and nobody can say what each spends.
- More than one provider is in use, each with its own keys, limits and dashboards.
- AI features are customer-facing and their cost must be attributed per customer or plan.
- A provider outage has already taken a feature down.
- A security or procurement review asks who can use which model with which data.
- A provider's model retirement surprised someone.
One team on one provider can do without. Several teams, providers or customers sharing AI spend, with someone accountable for it, cannot.
What a control plane is not
Clear boundaries make the category useful:
- Not a model provider. It runs no inference. If it also sells model access, that is a second business beside the control plane, with different incentives.
- Not an evaluation suite. It can record and sample answers, but datasets, experiments and prompt versioning are a development-lifecycle job; tools such as Langfuse do that beside the request.
- Not a data loss prevention system. Pattern and classifier checks on prompts reduce what reaches a provider; they do not replace your data governance.
- Not a place for business logic. Routing rules should express operational intent — which provider, which fallback, which budget — not product behaviour your application should own.
Build or buy
A minimal version — a proxy that holds keys and logs requests — is a few weeks of work. The parts that take longest are the ones that make it a control plane: budgets that hold under concurrent requests, per-model cost estimation across every pricing unit, provider-specific failover rules, an audit trail and the console people use to change policy. Build vs buy an LLM gateway sets out the engineering cost, and the enterprise AI gateway checklist lists what to evaluate.
INFRO's definition
INFRO describes itself as an AI control plane, and builds it on the BYOK model: you connect your own provider accounts, your providers bill you directly, and INFRO charges a software subscription with 0% markup on model spend. Its gateway is OpenAI-compatible and also answers Anthropic's Messages and OpenAI's Responses formats. Around it: sealed provider credentials and scoped gateway keys, routes with failover across your own connections, model allowlists at organization, project and key level, budgets at organization, project, member and key level checked before the provider is called, per-request cost estimates at list or negotiated rates, attribution by project, key, model, route and end customer, an audit log on every plan, and on the Business plan a monthly check of the estimate against OpenAI's and Anthropic's own cost APIs.
Summary
An AI control plane is the part of your AI stack that holds intent: keys, routes, access, budgets, attribution and audit, applied before a provider is called. Judge any product by which of those functions it performs, when it performs them, and whose provider accounts it runs on.