# What is an AI control plane? The layer between apps and models

URL: https://infro.io/blog/what-is-an-ai-control-plane
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-20
Last modified: 2026-10-20
Last reviewed: 2026-10-20 (next review due 2026-11-19)
Topic: Architecture and migration
Format: Guide

> An AI control plane is the layer between a company's applications and its AI providers that decides how requests run, separate from the providers that execute them. It holds provider credentials, routes and fails over requests, enforces model access and budgets before the provider is called, attributes cost and usage to projects and customers, and keeps an audit trail of who changed which policy. A gateway is usually its request path; the control plane is the policy, identity, cost and record keeping around it.

Key takeaways:
- A control plane decides how AI requests run; the providers that execute them are the data plane.
- Its core jobs: hold credentials, route and fail over, enforce model access and budgets before the call, attribute cost, and keep an audit trail.
- A gateway is usually its request path; an observability tool records but does not decide; an API gateway does not understand tokens or models.
- Whose provider accounts it runs on decides who owns your contracts, rates and exit.
- You need one when several teams, providers or customers share AI spend and someone has to answer for it.

**What is an AI control plane?** It is the layer between a company's applications and its AI providers that decides how requests run. It holds the provider credentials, routes and fails over requests, enforces which models each team may use and what each may spend before a provider is called, attributes cost and usage to projects and customers, and records who changed which policy. The providers that actually run the models are the data plane; the control plane is everything that decides, limits and records around them.

The term is borrowed from infrastructure, and the borrowing is precise. This guide explains what belongs in an AI control plane, how it differs from a gateway, an observability tool and an API gateway, and when a company needs one.

## Where the term comes from

In networking, the data plane forwards packets and the control plane decides where they should go. Kubernetes uses the same split: its control plane components "manage the overall state of the cluster", while node components run the workloads[1]. The control plane holds intent — what should happen — and the data plane does the work.

AI traffic now has the same shape. The work is done by model providers: OpenAI, Anthropic, Google, a cloud platform, a self-hosted model server. The intent — which provider, which model, which key, how much spend, whose request — has to live somewhere. In most companies it starts scattered across application code, environment variables and spreadsheets. An AI control plane puts it in one place.

## What an AI control plane does

| Function | What it decides or records | Without it |
| --- | --- | --- |
| Credentials | Which provider accounts exist, sealed; which keys applications use | Provider keys copied into services and laptops |
| Routing and reliability | Which provider and model serve a request; failover order; retries | Hard-coded model names; outages handled in each service |
| Access policy | Which models a project, team or key may call | Anyone with a key can call any model |
| Spend control | Estimated cost of each request; budgets that refuse before the call | Overspend discovered on the invoice |
| Attribution | Cost and usage by project, key, feature and end customer | One provider bill nobody can split |
| Audit | Who created keys, changed routes, raised budgets | No answer for the security review |
| Lifecycle | Which models are being retired and what still calls them | Shutdown dates found in production errors |

Anthropic's own documentation describes much of this as what a gateway gives an organization: provider keys kept server-side with developers holding gateway credentials, usage attributed per developer or team, budgets and rate limits enforced in one place, audit logging, and provider switching without touching developer machines[2]. The control plane is that list treated as a product, across every provider and application rather than one tool.

### Before the call, not after

The defining property is timing. A control plane acts **before** the provider is called: it can refuse a request from a key that may not use a model, or one that would exceed a budget, so the cost and the risk never occur. Recording what happened afterwards is necessary, but it is observability, not control. Several products in this market now enforce spend before the call; Cloudflare's AI Gateway, for example, blocks requests with a 429 once a spend limit is reached, and is explicit that its cost tracking is a best-effort estimate[3]. Every control plane works from estimates; the useful ones say so and check them against the provider's bill.

## Control plane, gateway, observability, API gateway

| Layer | In the request path? | Understands models and tokens? | Decides before the call? | Typical scope |
| --- | --- | --- | --- | --- |
| API gateway | Yes | No | Auth, rate limits | All HTTP APIs |
| LLM gateway | Yes | Yes | Routing, retries, sometimes limits | AI requests |
| Observability tool | No (traces sent to it) | Yes | No | Records and evaluation |
| AI control plane | Yes, through its gateway | Yes | Access, budgets, routing | Policy, cost and audit across an organization |

The lines blur because products combine layers. The useful question about any product is not what it calls itself but which of the functions in the first table it performs, and whether it performs them before or after the provider call. [What is an LLM gateway](https://infro.io/blog/what-is-an-llm-gateway) covers the request-path part in depth.

## Whose accounts does it run on?

The most consequential design choice in an AI control plane is commercial, not technical: whose provider accounts carry the traffic.

- **The platform's accounts.** The platform buys model access and charges you for it. You get one supplier relationship, and you lose your own contracts, negotiated rates, data agreements and the ability to leave without changing who runs your inference.
- **Your accounts (BYOK).** You connect provider accounts you already hold. Your providers bill you directly at your own rates; the control plane governs the traffic without sitting in the money flow.

Many products offer both, and the details matter: what happens when your own key fails (does the request fall back to the platform's capacity?) and whether budgets count spend on your own keys. [What is BYOK for AI](https://infro.io/blog/what-is-byok-ai) explains the model and the questions to ask.

## When a company needs one

Signs that the layer is overdue:

1. More than one team calls AI providers, and nobody can say what each spends.
2. More than one provider is in use, each with its own keys, limits and dashboards.
3. AI features are customer-facing and their cost must be attributed per customer or plan.
4. A provider outage has already taken a feature down.
5. A security or procurement review asks who can use which model with which data.
6. A provider's model retirement surprised someone.

One team on one provider can do without. Several teams, providers or customers sharing AI spend, with someone accountable for it, cannot.

## What a control plane is not

Clear boundaries make the category useful:

- **Not a model provider.** It runs no inference. If it also sells model access, that is a second business beside the control plane, with different incentives.
- **Not an evaluation suite.** It can record and sample answers, but datasets, experiments and prompt versioning are a development-lifecycle job; tools such as Langfuse do that beside the request.
- **Not a data loss prevention system.** Pattern and classifier checks on prompts reduce what reaches a provider; they do not replace your data governance.
- **Not a place for business logic.** Routing rules should express operational intent — which provider, which fallback, which budget — not product behaviour your application should own.

## Build or buy

A minimal version — a proxy that holds keys and logs requests — is a few weeks of work. The parts that take longest are the ones that make it a control plane: budgets that hold under concurrent requests, per-model cost estimation across every pricing unit, provider-specific failover rules, an audit trail and the console people use to change policy. [Build vs buy an LLM gateway](https://infro.io/blog/build-vs-buy-llm-gateway) sets out the engineering cost, and the [enterprise AI gateway checklist](https://infro.io/blog/enterprise-ai-gateway-checklist) lists what to evaluate.

## INFRO's definition

INFRO describes itself as an AI control plane, and builds it on the BYOK model: you connect your own provider accounts, your providers bill you directly, and INFRO charges a software subscription with 0% markup on model spend. Its gateway is OpenAI-compatible and also answers Anthropic's Messages and OpenAI's Responses formats. Around it: sealed provider credentials and scoped gateway keys, routes with failover across your own connections, model allowlists at organization, project and key level, budgets at organization, project, member and key level checked before the provider is called, per-request cost estimates at list or negotiated rates, attribution by project, key, model, route and end customer, an audit log on every plan, and on the Business plan a monthly check of the estimate against OpenAI's and Anthropic's own cost APIs.

## Summary

An AI control plane is the part of your AI stack that holds intent: keys, routes, access, budgets, attribution and audit, applied before a provider is called. Judge any product by which of those functions it performs, when it performs them, and whose provider accounts it runs on.

## Questions

### What is the difference between an AI gateway and an AI control plane?

A gateway is the endpoint requests pass through: it forwards, retries and fails over. A control plane is the policy and record keeping around that path: which keys exist, which models each team may use, what each may spend, how cost is attributed and who changed what. Many products combine both; the question is how much of the second part they cover.

### Is an AI control plane the same as LLMOps?

They overlap. LLMOps usually covers the development lifecycle: prompts, evaluation, deployment of models. An AI control plane covers the operation of AI requests across an organization: credentials, routing, access, spend, attribution and audit. Evaluation and prompt tools sit beside it.

### Do small teams need an AI control plane?

Not at first. One team calling one provider can keep a key in a secret manager and read the provider's dashboard. The need appears with several teams, several providers, customer-facing AI features whose cost must be attributed, or a security review asking who can use which model.

## Sources

1. Kubernetes Components — Kubernetes. https://kubernetes.io/docs/concepts/overview/components/ (accessed 2026-10-20)
2. Other LLM gateways — Anthropic. https://code.claude.com/docs/en/llm-gateway (accessed 2026-10-20)
3. Spend limits — Cloudflare. https://developers.cloudflare.com/ai-gateway/features/spend-limits/ (accessed 2026-10-20)