Teams that ship AI features tend to write the same code. A retry loop after the first rate limit. A second client after the second provider. A spreadsheet after the first surprising invoice. A list of "approved" models in a wiki that nothing enforces. None of it is the product, and all of it gets rewritten in the next service.
An LLM gateway is where that code goes instead. This guide explains what one does, what happens to a request inside it, what it should never do, and how to evaluate one before it sits between your users and your providers.
What an LLM gateway is
An LLM gateway — also called an AI gateway or an LLM proxy — is a network service that applications call instead of calling model providers directly. It exposes one API, commonly the shape of OpenAI's Chat Completions endpoint1, so an application already using an OpenAI client can switch by changing its base URL. Behind that API it holds or references provider credentials, decides which provider account serves each request, and records what happened.
It is easy to confuse with neighbouring tools, so it helps to be precise about the boundary:
| Tool | Where it runs | What it does | What it does not do |
|---|---|---|---|
| Provider SDK | In your process | Calls one provider's API | Fail over to another provider, or attribute cost across them |
| HTTP load balancer | In the network | Spreads traffic across healthy backends | Understand models, tokens, streaming semantics or cost |
| Orchestration framework | In your process | Chains prompts, tools and memory | Enforce policy for other services, or hold credentials centrally |
| Observability SDK | In your process | Records calls and traces | Refuse a request, route it, or retry it elsewhere |
| LLM gateway | In the request path | Routes, fails over, meters and enforces policy for every caller | Decide what your prompts should say or which answer is correct |
The four problems it absorbs
Provider interfaces differ in ways that matter
Providers expose similar capabilities through different contracts. Anthropic's Messages API, for example, takes the system prompt as a top-level system parameter, takes the output limit as max_tokens, and accepts message content either as a string or as an array of typed content blocks2, while the Chat Completions shape carries the system prompt as a message1. Error semantics differ too: Anthropic documents a distinct overloaded_error status alongside ordinary rate limiting3. Each difference is small. Handled separately in every service, they become the reason nobody wants to try a second provider.
Limits and failures belong to accounts, not to your application
Rate limits are set per organization and project and depend on the account's usage tier4. When a limit is hit the provider answers 429 Too Many Requests, which the HTTP specification allows to carry a Retry-After hint5. A launch, a batch job and a production feature that share one account compete for the same allowance, and a single account's incident is every dependent feature's incident.
Cost is invisible at the level decisions are made
A provider invoice tells you what an account spent. It does not tell you which feature, customer, prompt version or agent loop spent it. Every response carries token counts, but turning those into cost per workload — at the rate your contract actually carries — is work that lands on whoever is asked why the bill moved.
Policy has no single place to live
"Only these models in production", "this key may not exceed this budget" and "prompts are not stored" are organizational decisions. Without a shared enforcement point they are conventions, and conventions are copied inconsistently into every repository that holds a provider key. Unbounded consumption — runaway loops and uncapped usage — is listed as its own risk category for LLM applications6. The same applies to content rules such as "no card numbers or API keys in prompts": checked in one place in the request path, they hold for every service at once, which is the case made in guardrails at the gateway.
What happens to a request inside a gateway
The sequence is short, and a good gateway lets you inspect every step of it after the fact.
Your application
INFRO gateway
- 1AuthenticateGateway key, its project and its environment.
- 2Apply policyModel allowlists; reserve the request's worst-case cost against any blocking budget.
- 3ResolveA route slug becomes its ordered targets; a model id becomes the connections that serve its vendor.
- 4AttemptSend to a target. Fail over to the next only before the response has started.
- 5RecordModel, connection, every failed attempt, latency, token counts and estimated provider cost.
Your provider accounts
- Authenticate. The application sends a normal API request with a gateway key. The key identifies the organization, the project and the environment the request belongs to.
- Apply policy. Before any provider is contacted, the gateway checks what the key is allowed to do: which models it may call, whether a budget would be exceeded by the worst case of this request, and, where content checks are configured, whether the prompt may be sent as it is. A gateway with an opt-in response cache can also answer an identical earlier request here, without contacting a provider at all.
- Resolve. The requested model name becomes an ordered list of candidates. That might be a named route with explicit targets — chosen, in some gateways, by rules that match on request metadata — or every connected account that can serve the model.
- Attempt. The gateway sends the request to the first candidate in that provider's own wire format. If the attempt fails before any response has been sent back, it can move to the next candidate.
- Relay. For streaming responses it forwards server-sent events as they arrive7 rather than buffering the whole answer.
- Record. When the request finishes — successfully or not — it writes one record: which model and account served it, each failed attempt, latency, token counts and the estimated cost.
The distinction in step 4 is the one most worth understanding: once the first part of a response has reached the client, moving to another provider would produce duplicated or contradictory output. A gateway that retries after that point is hiding a failure, not recovering from one.
Three credential models, and why they matter more than features
Before comparing capabilities, find out who holds the provider accounts. It decides who invoices you for model usage, whose negotiated rates apply, which parties can see your data, and how you leave.
| Your accounts, hosted gateway | Vendor's accounts, hosted gateway | Your accounts, self-hosted proxy | |
|---|---|---|---|
| Who holds the provider contract | You | The gateway vendor | You |
| Who invoices model usage | Your providers, directly | The gateway vendor | Your providers, directly |
| Your negotiated rates and commitments | Apply | Replaced by the vendor's pricing | Apply |
| Parties in the data path | Gateway vendor and your providers | Gateway vendor and its providers | Your infrastructure and your providers |
| Who runs and pages for the gateway | The vendor | The vendor | You |
| Leaving | Point the base URL back at your providers | Re-contract with each provider | Replace a service you operate |
Open-source proxies such as LiteLLM show that the third model is practical when a team is prepared to operate a service in its request path8. The first model trades that operational work for a vendor dependency while keeping the commercial relationship with each provider where it already is.
The line between the first two columns is blurrier than the table suggests, because several hosted gateways accept your keys and then fall back to their own accounts when your key fails. OpenRouter documents that when your keys hit a rate limit or fail, it falls back to its shared endpoints by default, and lets you turn that off per key9. Vercel's AI Gateway documents that a failed request on your credentials is retried with its system credentials and billed against your balance with Vercel10. Both are reasonable availability choices; both mean some of your traffic can run on a contract that is not yours unless you change the default. Who owns the gateway vendor matters too: several changed hands in 2026, covered in who owns your AI gateway.
What a gateway costs you
Adopting one is not free, and evaluating one honestly means measuring these before production:
- Latency. It is one more network hop and one more TLS connection pool. Measure time to first token with and without it, from the regions your application runs in, at your real concurrency.
- A party in your data path. Prompts and outputs transit the gateway. Read its retention defaults, its subprocessor list and what it stores when payload capture is off.
- Feature lag. A provider's newest parameter may not pass through on day one. Check how unknown parameters are handled: forwarded, dropped, or rejected.
- A new dependency. The gateway's availability is now part of yours. Keep the exit cheap: if your application uses a standard client with a configurable base URL, reverting is a configuration change.
When you do not need one
One application, one provider, modest volume, no requirement to attribute cost below the account, and no policy to enforce: call the provider directly. A gateway starts paying for itself when a second provider, a second team, a budget owner or a security review arrives — usually in that order.
How to evaluate a gateway
Run these against a real workload in a test environment. Every item should end in something you observed, not something you were told.
- Streaming, tool calls, structured outputs and cancellations behave the same through the gateway as directly.
- A forced provider failure produces the behaviour you expect, and the request record shows the failed attempt.
- A failure after streaming has started is surfaced as an error, not silently restarted on another provider.
- Each record names the provider and account that served the request, its latency and its estimated cost.
- A request for a model with no known rate is recorded as unpriced, not as free.
- When your own provider key fails, you know whether the request moves to another of your accounts, fails, or runs on the vendor's account — and you chose that behaviour.
- Budgets can both alert and block, at more than one scope, and the refusal names what refused.
- Budgets count spend on your own provider keys, not only spend through the vendor's accounts (LLM spend limits compared shows how much this varies).
- Provider credentials are encrypted at rest, scoped to your organization and never displayed after saving.
- Prompt and completion capture is off by default, with a retention window you choose.
- Request records are retrievable through an API, so your history is not trapped in a console.
How INFRO implements the pattern
INFRO is an LLM gateway on the first credential model. You connect provider accounts you already hold; applications call one OpenAI-compatible endpoint with an INFRO gateway key. The change to an existing OpenAI client is the base URL, the key and the model id:
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.infro.io/v1",
api_key=os.environ["INFRO_API_KEY"], # a gateway key, not a provider key
)
response = client.chat.completions.create(
model="support-chat", # a route you defined, or a model id such as "anthropic/claude-sonnet-5"
messages=[{"role": "user", "content": "Summarize this ticket thread."}],
)The response keeps the Chat Completions shape and adds the fields that say where the request ran. This example is abbreviated and its identifiers are illustrative:
{
"id": "req_…",
"object": "chat.completion",
"model": "anthropic/claude-sonnet-5",
"route": "primary",
"route_name": "support-chat",
"provider": "anthropic",
"connection": "conn_…",
"usage": { "prompt_tokens": 812, "completion_tokens": 164, "total_tokens": 976, "cost": 0.004896 }
}usage.cost is an estimate of what your provider will bill for the request, from your negotiated rate when you have entered one and the model vendor's list rate otherwise; it is null when no rate is known. The same request, with any failed attempts, is retrievable later from GET /v1/requests/{id}. See the quickstart and provider connections for the full contract.
A failed attempt moves to the next of your own connections; INFRO has no accounts of its own to fall back to, so when every target fails the request fails, and for provider failures the error message names each connection tried. Routes can carry if-rules that pick targets from request metadata and failure rules that decide, per failure kind, whether to retry, move on or stop — see routing rules and failure rules. Two other steps from the request path above are opt-in: an exact-match response cache for chat completions, off by default, whose hits never reach your provider; and guardrails that block, redact or flag patterns such as card numbers and API keys in prompts and responses, included from the Business plan.
It is equally important to be clear about what INFRO does not do today: the response cache matches identical requests only, never similar ones; it does not re-route a streaming response after the first event has been sent; and it publishes no uptime SLA on self-serve plans.