Gateway down: should AI traffic fail open or fail closed?

What the gateway was doing for each request

Before choosing a failure mode, list what disappears when the gateway does. A gateway in front of every provider call usually does five jobs per request: authenticates the caller, admits the request against a budget, applies policy (model allowlists, guardrails), picks a provider connection, and records what happened. Provider failover is the sixth job, and it does not help here: connection failover and model fallbacks operate on upstream failures before the first token, and the gateway's own refusals are decided before any provider is called and return without failover¹. If the gateway process is the thing that is down, none of that logic runs.

That is why "fail open" is a misleading name for a single behavior. Fail-open means one job survives (getting bytes to a provider) while four are dropped. Fail-closed keeps all four by refusing the request. The useful question is which of the dropped jobs you can afford to lose for this workload, for this long.

Three modes and what each does

Most production setups end up with all three, assigned per workload class. The rest of this article explains the rows that teams tend to underestimate: budgets, audit and keys.

Budgets: the cap only holds if the request passes through it

A blocking budget is a pre-call check. INFRO's, for example, admits a request only if its worst-case estimated cost fits the room left, counting recorded spend plus reservations held across its infrastructure². That check cannot run on a request that never reaches the gateway. A bypass therefore moves spend into a place your gateway budget does not see, and the provider's own limits (or none) are all that stands between a bad loop and the invoice.

The gateway's own design shows the safer default. When INFRO cannot reach its database in time to admit a request, it refuses it with 503, code gateway_unavailable, and a Retry-After header; the request is never served unchecked²³. That is fail-closed applied to budget admission: an unavailable limiter refuses rather than waves traffic through. If your own bypass logic is more permissive than the gateway's internal behavior, you have chosen to be less careful than the component you are routing around.

If you do bypass, spend limits set at the provider on the break-glass key are the only ceiling. Check what your provider's limits actually stop before relying on them; AI provider spend limits compared covers that question. Afterwards, find the bypassed usage with the method in Detect AI API usage that bypassed your gateway.

Keys: fail-open is a credential decision

In a BYOK gateway design, provider credentials sit inside the gateway and applications hold only a gateway key. INFRO's documentation states this directly: provider credentials stay sealed, and applications only ever hold the INFRO key⁴. A direct-to-provider fallback inverts that. For the bypass to work at 3 a.m., the application, a sidecar or a secrets-manager path must already be able to produce a provider key.

There are three ways to arrange it, in rising order of risk:

1. Break-glass keys fetched on demand. A human or an automation with approval retrieves a dedicated provider key, scoped to one project and capped at the provider, and injects it only for the incident. Slow, but the key does not sit in application config.
2. Dormant keys in the secrets manager. The key exists, is readable by a narrow identity, and is exercised in drills. Faster, with a standing credential to inventory and rotate.
3. Provider keys in application config. Always available, and exactly the exposure the gateway removed. Every deployment, log line and crash dump now touches a credential that can bill your account.

Whichever you choose, put the break-glass credential in your credential inventory, rotate it after each use, and alert on provider-side usage from it. A key that can only be used during an incident should produce a loud signal whenever it is used outside one.

Audit: bypassed calls still need a record

Gateway request logs are the usual evidence of who called which model with what outcome. In a bypass those records stop. Do not assume telemetry fills the gap: INFRO's OpenTelemetry export, for instance, is best effort, runs outside the response path, and drops spans on persistent failure instead of blocking requests; its docs say not to use it as the sole accounting record⁵. A telemetry pipeline is for observation, not for proving what happened.

The practical answer is a local journal written by whatever performs the bypass. Record at least: timestamp, caller identity, workload class, provider, model, a request ID you generate, token or unit counts from the response, and a flag marking the call as bypassed. Then apply the same durability thinking that OpenTelemetry's Collector documentation recommends for telemetry hops: a sending queue with retry and backoff, and a write-ahead log on disk so a crash does not lose the buffer⁶. That page also lists how loss still happens (queue overflow, retries exhausted, no persistence), which is a good checklist for your journal: size it for the longest outage you plan to survive, and monitor its depth.

After recovery, replay the journal into the same store as normal request records, tagged as bypassed, and reconcile it against provider usage reports. Gateway estimates and provider invoices drift anyway, so the journal also gives the finance team a labeled reason for any gap in that period.

Fail-closed done properly

Failing closed is not "throw the error to the user." It is a client behavior you design. The status and code are the signal: INFRO documents gateway_unavailable as a 503³, and its guidance is that 408, 429, 502 and 503 are the retryable statuses while 400, 401, 402 and 404 should never be retried automatically⁷. Honor Retry-After, add jitter, and cap total retries so a recovering gateway is not hit by every client at once.

import random
import time

import httpx

RETRYABLE = {408, 429, 502, 503}
NEVER_RETRY = {400, 401, 402, 404}

# Mode per workload class; reviewed by the service owner, not guessed at 3 a.m.
POLICY = {
    "billing-extraction": "closed",   # spends money, needs an audit record
    "agent-tools":        "closed",   # side effects, unbounded loops
    "support-suggest":    "degraded", # fall back to canned text
    "autocomplete":       "open",     # tiny calls, low risk
}

def call(workload: str, payload: dict, max_attempts: int = 3):
    mode = POLICY[workload]
    for attempt in range(max_attempts):
        resp = httpx.post(GATEWAY_URL, json=payload, headers=AUTH, timeout=30)
        if resp.status_code < 400:
            return resp.json()
        if resp.status_code in NEVER_RETRY:
            resp.raise_for_status()
        if resp.status_code in RETRYABLE:
            wait = float(resp.headers.get("Retry-After", 2 ** attempt))
            time.sleep(wait + random.uniform(0, 0.5))
            continue
        resp.raise_for_status()
    # Gateway still unavailable after bounded retries
    if mode == "open" and BYPASS_ENABLED:      # a flag a human or runbook sets
        return bypass_call(workload, payload)  # journals every call
    if mode == "degraded":
        return enqueue_or_canned(workload, payload)
    raise GatewayUnavailable(workload)

Two details matter. The bypass is behind an explicit flag, so a transient 503 does not silently route spend around the controls. And a 402 is never retried or bypassed: it means a budget said no, and going around it defeats the budget.

Degraded modes for traffic that can wait

Between refusing and bypassing there is a lot of room. Four patterns work without touching provider keys:

• Queue and replay for asynchronous work such as batch classification, report generation or media jobs. The user sees "processing," and the work runs under the gateway's budgets when it returns.
• Load shedding of optional features. Drop the "suggested reply" before dropping checkout assistance.
• Static or cached answers held on the application side, for well-known questions.
• Reduced scope, such as disabling tool-calling agents while keeping plain chat, if you have a bypass for chat only.

Degraded modes are also where the policy for agents belongs. An agent loop that retries through an outage burns its step budget and can repeat side effects; give it a hard stop on the first 503 after its bounded retries rather than a bypass. See Bounding AI agent loops for step, time and spend limits.

Choosing per workload

A workable rule: fail closed unless all of these hold, in which case fail open is defensible.

• The per-call cost is low and the call rate is bounded by the caller, not by model output or loops.
• No regulated or sensitive data is sent, or the client applies the same redaction the gateway would.
• No audit requirement attaches to the individual call, or a local journal satisfies it.
• A capped, dedicated provider credential exists and has been exercised in a drill.
• Someone owns the decision to throw the switch and the decision to throw it back.

For everything else, choose closed or degraded, and write the choice down next to the workload in your service catalog. Then run it: Failover drills describes how to rehearse provider failure, and the same exercise should include a deliberate gateway outage, a flip to bypass, a journal replay and a key rotation.

How INFRO treats the closed case

INFRO's budget admission is fail-closed, as described above, and it returns gateway_unavailable as a distinct 503 your clients can branch on³. Its spend controls (opens in a new tab) cover organization, project, member and key limits, and provider connections stay sealed behind gateway keys that can be limited and rotated with an overlap window. Request records, the audit log and OpenTelemetry export give you the records a bypass journal has to be reconciled against. None of those can protect traffic that goes around the gateway, which is why the bypass design above stays your decision.

Checklist before you need it

• Every workload has a written mode: closed, open or degraded.
• Clients branch on 503, honor Retry-After, add jitter and cap retries.
• Bypass is behind an explicit flag with a named owner.
• Break-glass provider keys are separate, capped at the provider and inventoried.
• A local journal with a persistent queue records every bypassed call.
• A reconciliation step compares the journal with provider usage after the outage.
• The bypass and the return to normal were both exercised in a drill this quarter.