# Gateway down: should AI traffic fail open or fail closed?

URL: https://infro.io/blog/gateway-down-should-ai-traffic-fail-open-or-fail-closed
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-04
Last modified: 2026-11-04
Last reviewed: 2026-11-04 (next review due 2027-05-03)
Topic: Architecture and migration
Format: Analysis

> Fail closed by default for anything that spends money, touches regulated data or needs an audit record. Fail open only for a short list of low-risk workloads, with break-glass provider keys that stay capped, dormant and journaled. Most teams end up with a per-workload mix plus a degraded mode.

Key takeaways:
- A gateway outage removes budget admission, policy, routing and recording at once; fail-open keeps only routing.
- Fail-open requires provider credentials outside the gateway, which is the main security cost of the mode.
- Spend and audit gaps from a bypass are real: journal every bypassed call locally and reconcile with provider usage afterwards.
- Choose the mode per workload class, not per company, and add a degraded mode (queue, shed, static answer) for traffic that can wait.
- Drill the switch, because a bypass path nobody has exercised is usually broken when it is needed.

## What the gateway was doing for each request

Before choosing a failure mode, list what disappears when the gateway does. A gateway in front of every provider call usually does five jobs per request: authenticates the caller, admits the request against a budget, applies policy (model allowlists, guardrails), picks a provider connection, and records what happened. Provider failover is the sixth job, and it does not help here: connection failover and model fallbacks operate on upstream failures before the first token, and the gateway's own refusals are decided before any provider is called and return without failover[1]. If the gateway process is the thing that is down, none of that logic runs.

That is why "fail open" is a misleading name for a single behavior. Fail-open means one job survives (getting bytes to a provider) while four are dropped. Fail-closed keeps all four by refusing the request. The useful question is which of the dropped jobs you can afford to lose for this workload, for this long.

## Three modes and what each does

> **Note: Definitions used here**
> **Fail-open**: the client or a sidecar sends the request straight to the provider, skipping the gateway. **Fail-closed**: the client gets an error and the request is not made. **Degraded**: the request is held, shed, or answered by something that needs neither the gateway nor a provider.

| Property | Fail-closed | Fail-open (direct bypass) | Degraded (queue, shed, static answer) |
| --- | --- | --- | --- |
| User-visible result | Error or retry delay | Normal answers | Delayed, reduced or canned answers |
| Budget enforcement | Intact; nothing spends | Gone; gateway caps cannot see the spend | Intact if work is queued until the gateway returns |
| Policy and guardrails | Intact | Skipped unless duplicated in the client | Intact for queued work |
| Audit and request log | Complete; refused calls can be logged client-side | Gap unless journaled locally | Complete once replayed |
| Provider key exposure | None added | Provider credentials must live outside the gateway | None added |
| Failure to watch for | Retry storms against a recovering gateway | Unmetered spend, unreviewed traffic, keys left in place | Queue growth, stale answers |
| Fits | Spend-bearing, regulated or agentic workloads | Low-risk, user-facing, low-cost calls | Batch, async and background work |

Table: Effect of each mode on the controls a gateway provides.

Most production setups end up with all three, assigned per workload class. The rest of this article explains the rows that teams tend to underestimate: budgets, audit and keys.

## Budgets: the cap only holds if the request passes through it

A blocking budget is a pre-call check. INFRO's, for example, admits a request only if its worst-case estimated cost fits the room left, counting recorded spend plus reservations held across its infrastructure[2]. That check cannot run on a request that never reaches the gateway. A bypass therefore moves spend into a place your gateway budget does not see, and the provider's own limits (or none) are all that stands between a bad loop and the invoice.

The gateway's own design shows the safer default. When INFRO cannot reach its database in time to admit a request, it refuses it with `503`, code `gateway_unavailable`, and a `Retry-After` header; the request is never served unchecked[2][3]. That is fail-closed applied to budget admission: an unavailable limiter refuses rather than waves traffic through. If your own bypass logic is more permissive than the gateway's internal behavior, you have chosen to be less careful than the component you are routing around.

> **Example: Illustrative bypass arithmetic**
> These numbers are made up to show the shape of the exposure. Suppose a feature sends 40 requests per second and a typical call costs an assumed $0.004 at the provider. A 30-minute bypass is 40 × 1,800 = 72,000 requests, roughly $288 of spend that no gateway budget, key limit or per-customer cap observed. Now imagine the same outage coinciding with a runaway agent loop: the cap that would have stopped it is the one component that is down.

If you do bypass, spend limits set at the provider on the break-glass key are the only ceiling. Check what your provider's limits actually stop before relying on them; [AI provider spend limits compared](https://infro.io/blog/provider-spend-limits-compared) covers that question. Afterwards, find the bypassed usage with the method in [Detect AI API usage that bypassed your gateway](https://infro.io/blog/detect-gateway-bypass-provider-usage).

## Keys: fail-open is a credential decision

In a BYOK gateway design, provider credentials sit inside the gateway and applications hold only a gateway key. INFRO's documentation states this directly: provider credentials stay sealed, and applications only ever hold the INFRO key[4]. A direct-to-provider fallback inverts that. For the bypass to work at 3 a.m., the application, a sidecar or a secrets-manager path must already be able to produce a provider key.

There are three ways to arrange it, in rising order of risk:

1. **Break-glass keys fetched on demand.** A human or an automation with approval retrieves a dedicated provider key, scoped to one project and capped at the provider, and injects it only for the incident. Slow, but the key does not sit in application config.
2. **Dormant keys in the secrets manager.** The key exists, is readable by a narrow identity, and is exercised in drills. Faster, with a standing credential to inventory and rotate.
3. **Provider keys in application config.** Always available, and exactly the exposure the gateway removed. Every deployment, log line and crash dump now touches a credential that can bill your account.

Whichever you choose, put the break-glass credential in your credential inventory, rotate it after each use, and alert on provider-side usage from it. A key that can only be used during an incident should produce a loud signal whenever it is used outside one.

## Audit: bypassed calls still need a record

Gateway request logs are the usual evidence of who called which model with what outcome. In a bypass those records stop. Do not assume telemetry fills the gap: INFRO's OpenTelemetry export, for instance, is best effort, runs outside the response path, and drops spans on persistent failure instead of blocking requests; its docs say not to use it as the sole accounting record[5]. A telemetry pipeline is for observation, not for proving what happened.

The practical answer is a local journal written by whatever performs the bypass. Record at least: timestamp, caller identity, workload class, provider, model, a request ID you generate, token or unit counts from the response, and a flag marking the call as bypassed. Then apply the same durability thinking that OpenTelemetry's Collector documentation recommends for telemetry hops: a sending queue with retry and backoff, and a write-ahead log on disk so a crash does not lose the buffer[6]. That page also lists how loss still happens (queue overflow, retries exhausted, no persistence), which is a good checklist for your journal: size it for the longest outage you plan to survive, and monitor its depth.

After recovery, replay the journal into the same store as normal request records, tagged as bypassed, and reconcile it against provider usage reports. Gateway estimates and provider invoices drift anyway, so the journal also gives the finance team a labeled reason for any gap in that period.

## Fail-closed done properly

Failing closed is not "throw the error to the user." It is a client behavior you design. The status and code are the signal: INFRO documents `gateway_unavailable` as a `503`[3], and its guidance is that `408`, `429`, `502` and `503` are the retryable statuses while `400`, `401`, `402` and `404` should never be retried automatically[7]. Honor `Retry-After`, add jitter, and cap total retries so a recovering gateway is not hit by every client at once.

```python
import random
import time

import httpx

RETRYABLE = {408, 429, 502, 503}
NEVER_RETRY = {400, 401, 402, 404}

# Mode per workload class; reviewed by the service owner, not guessed at 3 a.m.
POLICY = {
    "billing-extraction": "closed",   # spends money, needs an audit record
    "agent-tools":        "closed",   # side effects, unbounded loops
    "support-suggest":    "degraded", # fall back to canned text
    "autocomplete":       "open",     # tiny calls, low risk
}

def call(workload: str, payload: dict, max_attempts: int = 3):
    mode = POLICY[workload]
    for attempt in range(max_attempts):
        resp = httpx.post(GATEWAY_URL, json=payload, headers=AUTH, timeout=30)
        if resp.status_code < 400:
            return resp.json()
        if resp.status_code in NEVER_RETRY:
            resp.raise_for_status()
        if resp.status_code in RETRYABLE:
            wait = float(resp.headers.get("Retry-After", 2 ** attempt))
            time.sleep(wait + random.uniform(0, 0.5))
            continue
        resp.raise_for_status()
    # Gateway still unavailable after bounded retries
    if mode == "open" and BYPASS_ENABLED:      # a flag a human or runbook sets
        return bypass_call(workload, payload)  # journals every call
    if mode == "degraded":
        return enqueue_or_canned(workload, payload)
    raise GatewayUnavailable(workload)
```

Two details matter. The bypass is behind an explicit flag, so a transient `503` does not silently route spend around the controls. And a `402` is never retried or bypassed: it means a budget said no, and going around it defeats the budget.

## Degraded modes for traffic that can wait

Between refusing and bypassing there is a lot of room. Four patterns work without touching provider keys:

- **Queue and replay** for asynchronous work such as batch classification, report generation or media jobs. The user sees "processing," and the work runs under the gateway's budgets when it returns.
- **Load shedding** of optional features. Drop the "suggested reply" before dropping checkout assistance.
- **Static or cached answers** held on the application side, for well-known questions.
- **Reduced scope**, such as disabling tool-calling agents while keeping plain chat, if you have a bypass for chat only.

Degraded modes are also where the policy for agents belongs. An agent loop that retries through an outage burns its step budget and can repeat side effects; give it a hard stop on the first `503` after its bounded retries rather than a bypass. See [Bounding AI agent loops](https://infro.io/blog/bound-ai-agent-loops) for step, time and spend limits.

## Choosing per workload

A workable rule: fail closed unless all of these hold, in which case fail open is defensible.

- The per-call cost is low and the call rate is bounded by the caller, not by model output or loops.
- No regulated or sensitive data is sent, or the client applies the same redaction the gateway would.
- No audit requirement attaches to the individual call, or a local journal satisfies it.
- A capped, dedicated provider credential exists and has been exercised in a drill.
- Someone owns the decision to throw the switch and the decision to throw it back.

For everything else, choose closed or degraded, and write the choice down next to the workload in your service catalog. Then run it: [Failover drills](https://infro.io/blog/llm-failover-drills) describes how to rehearse provider failure, and the same exercise should include a deliberate gateway outage, a flip to bypass, a journal replay and a key rotation.

## How INFRO treats the closed case

INFRO's budget admission is fail-closed, as described above, and it returns `gateway_unavailable` as a distinct `503` your clients can branch on[3]. Its [spend controls](https://infro.io/docs/org/spend-controls) cover organization, project, member and key limits, and provider connections stay sealed behind gateway keys that can be limited and rotated with an overlap window. Request records, the audit log and OpenTelemetry export give you the records a bypass journal has to be reconciled against. None of those can protect traffic that goes around the gateway, which is why the bypass design above stays your decision.

## Checklist before you need it

- [ ] Every workload has a written mode: closed, open or degraded.
- [ ] Clients branch on `503`, honor `Retry-After`, add jitter and cap retries.
- [ ] Bypass is behind an explicit flag with a named owner.
- [ ] Break-glass provider keys are separate, capped at the provider and inventoried.
- [ ] A local journal with a persistent queue records every bypassed call.
- [ ] A reconciliation step compares the journal with provider usage after the outage.
- [ ] The bypass and the return to normal were both exercised in a drill this quarter.

## Questions

### Should the fail-open path use the same provider keys the gateway holds?

Prefer separate break-glass keys. They can be scoped to one project, capped at the provider, kept out of application config until the switch is thrown, and rotated after every use. Reusing the gateway's own credentials means copying them into places the gateway was meant to keep them out of.

### How long should an outage last before traffic flips to bypass?

Pick a threshold from the business cost of failed requests and the cost of unmetered spend, not from a universal number. Many teams use a short automated retry window for fail-closed traffic and require a human decision for any bypass of workloads that carry spend or sensitive data.

## Sources

1. Failover & fallbacks — INFRO Docs. https://infro.io/docs/features/fallbacks (accessed 2026-11-04)
2. Spend controls — INFRO Docs. https://infro.io/docs/org/spend-controls (accessed 2026-11-04)
3. Errors — INFRO Docs. https://infro.io/docs/api/errors (accessed 2026-11-04)
4. Authentication — INFRO Docs. https://infro.io/docs/authentication (accessed 2026-11-04)
5. Exports & OpenTelemetry — INFRO Docs. https://infro.io/docs/observability/exports (accessed 2026-11-04)
6. Resiliency — OpenTelemetry. https://opentelemetry.io/docs/collector/resiliency/ (accessed 2026-11-04)
7. Rate & spend limits — INFRO Docs. https://infro.io/docs/api/limits (accessed 2026-11-04)