# Failover drills: test LLM provider failure before it happens

URL: https://infro.io/blog/llm-failover-drills
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Routing and reliability
Format: Methodology

> Run failover drills as small experiments: define the route's normal behaviour, state what should happen under one specific fault, inject that fault through a mock provider that speaks the real wire format, and verify the outcome from the request records rather than from the absence of errors. Cover faults before the first event, during a stream and across the whole chain, and repeat the drills on a schedule.

Failover code has an unusual property: it only runs when something else has already gone wrong. A retry loop, a fallback target, a stream-interruption handler — each sits unexecuted through months of healthy traffic, and then runs for the first time, at scale, during a provider incident. That is the worst possible moment to discover that the standby key was revoked, the timeout is longer than the client's deadline, or the error handler assumes every failure has a status code.

A drill runs that code on purpose, one fault at a time, while you are watching. The AWS Well-Architected reliability guidance lists "designing for resiliency, but not verifying" among the common anti-patterns it warns against, and recommends running fault-injection experiments regularly, as close to production as possible[1].

## Treat each drill as an experiment

The Principles of Chaos Engineering describe an experiment in four steps: define steady state as a measurable output of normal behaviour, hypothesize that it continues under the fault, introduce a real-world fault, and try to disprove the hypothesis[2]. That structure keeps a drill honest. Without a written hypothesis, any outcome can be read as success.

[Diagram: A failover drill as an experiment. The hypothesis is written before the fault is injected, and the drill has a stop condition.]

For a route serving a chat feature, steady state is a success rate, a p95 time to first token and a fallback share, taken from the request records before you start. A good hypothesis names the fault, the mechanism and the measurable outcome: *if the primary returns 503, the standby serves every request within the client's 30-second deadline, the fallback share rises, and no user receives an error.* The same principles ask you to minimize the blast radius and to automate experiments so they run continuously[2].

## Choose faults from how providers actually fail

A drill is only as useful as the faults it injects. Provider failures cluster into a few shapes, and each exercises a different code path.

[Diagram: Six faults worth drilling, where each happens relative to the response, and what should be observed. The first four should be invisible to users; the last two must be handled by the application.]

- **Status errors before any response.** A 429 with a wait hint, a 503, a 500, and on Anthropic's API a `529 overloaded_error`[3]. These test classification: which errors move to another target, and which do not.
- **Slow headers.** The provider accepts the connection and takes a long time to answer. This tests per-attempt timeouts against the caller's deadline.
- **Headers, then silence.** The stream opens and nothing arrives. This tests idle limits, which are usually longer and less examined.
- **A dropped stream.** Some content arrives and the connection closes without a terminal event. A partial event at the cut is discarded by the SSE parser, as the specification requires[4]; the drill checks what your application does with the text that did arrive.
- **Everything down.** Every target fails. This tests the error your application finally receives, and whether it handles it.

Use your own incident history to add to the list; post-incident analyses are the guidance's recommended source for which faults to experiment with[1].

## A mock upstream that fails on demand

The simplest reliable fault injector is a small server that speaks the provider's wire format and fails in whichever way it is told. It exercises your real client and gateway code, including parsing, instead of a mocked function. The server below implements the OpenAI-compatible chat completions endpoint, streaming and non-streaming, with a control endpoint to switch modes.

```typescript
import { createServer, type ServerResponse } from "node:http";

type Mode = "ok" | "429" | "503" | "slow_headers" | "silent_after_headers"
  | "drop_mid_stream" | "error_mid_stream";

let mode: Mode = (process.env.FAULT_MODE as Mode | undefined) ?? "ok";
const CONTROL_TOKEN = process.env.CONTROL_TOKEN ?? "";   // protects the mode switch
const UPSTREAM_KEY = process.env.UPSTREAM_KEY ?? "";     // the key the gateway sends
const CUT_AFTER = Number(process.env.CUT_AFTER ?? 3);
const SLOW_MS = Number(process.env.SLOW_MS ?? 30_000);

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
const now = () => Math.floor(Date.now() / 1000);

function send(res: ServerResponse, payload: unknown) {
  res.write(`data: ${JSON.stringify(payload)}\n\n`);
}

function chunk(delta: Record<string, string>, finish: string | null = null) {
  return {
    id: "chatcmpl-drill", object: "chat.completion.chunk", created: now(), model: "drill-model",
    choices: [{ index: 0, delta, finish_reason: finish }],
  };
}

function json(res: ServerResponse, status: number, body: unknown, headers: Record<string, string> = {}) {
  res.writeHead(status, { "content-type": "application/json", ...headers });
  res.end(JSON.stringify(body));
}

createServer(async (req, res) => {
  const url = new URL(req.url ?? "/", "http://drill.local");

  if (req.method === "POST" && url.pathname === "/__fault") {
    if (!CONTROL_TOKEN || req.headers.authorization !== `Bearer ${CONTROL_TOKEN}`) return json(res, 401, {});
    mode = (url.searchParams.get("mode") as Mode | null) ?? "ok";
    return json(res, 200, { mode });
  }
  if (req.method !== "POST" || !url.pathname.endsWith("/chat/completions")) return json(res, 404, {});
  if (UPSTREAM_KEY && req.headers.authorization !== `Bearer ${UPSTREAM_KEY}`) {
    return json(res, 401, { error: { message: "bad key", type: "invalid_request_error" } });
  }

  let raw = "";
  for await (const part of req) raw += part;
  const body = JSON.parse(raw || "{}") as { stream?: boolean };
  const words = ["Drill", " answer", " from", " the", " mock", " upstream."];

  if (mode === "429") {
    return json(res, 429, { error: { message: "Rate limit reached (drill)", type: "requests" } }, { "retry-after": "7" });
  }
  if (mode === "503") {
    return json(res, 503, { error: { message: "Temporarily overloaded (drill)", type: "server_error" } });
  }
  if (mode === "slow_headers") await sleep(SLOW_MS);         // then answer normally
  if (mode === "silent_after_headers") {
    res.writeHead(200, { "content-type": "text/event-stream" });
    res.flushHeaders();
    return;                                                   // never writes, never ends
  }

  if (!body.stream) {
    return json(res, 200, {
      id: "chatcmpl-drill", object: "chat.completion", created: now(), model: "drill-model",
      choices: [{ index: 0, message: { role: "assistant", content: words.join("") }, finish_reason: "stop" }],
      usage: { prompt_tokens: 12, completion_tokens: 6, total_tokens: 18 },
    });
  }

  res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
  send(res, chunk({ role: "assistant", content: "" }));
  for (const [i, word] of words.entries()) {
    if (i === CUT_AFTER && mode === "drop_mid_stream") return void res.destroy();   // no terminator
    if (i === CUT_AFTER && mode === "error_mid_stream") {
      send(res, { error: { message: "Stream failed (drill)", type: "server_error" } });
      return void res.end();
    }
    send(res, chunk({ content: word }));
    await sleep(150);
  }
  send(res, chunk({}, "stop"));
  res.write("data: [DONE]\n\n");
  res.end();
}).listen(Number(process.env.PORT ?? 8787));
```

Point a client or SDK straight at it to test your own retry and stream handling in isolation. To drill a gateway, put it behind HTTPS and register it as a provider target, as below.

## Drilling a route in INFRO

INFRO lets you add the mock as an ordinary connection, so drills run through the same routing, failover and records as production traffic, without touching a production route.

[Diagram: The drill harness: a runner with a staging key, a staging route whose primary is the mock and whose standby is a real connection, and the request records as evidence.]

1. **Connect the mock** as an [OpenAI-compatible endpoint](https://infro.io/docs/providers): its HTTPS base URL, the key it expects, and a model map from the catalog model your route serves to `drill-model`. A model with no map entry is not served through that kind of connection. Scope the connection to the `staging` environment.
2. **Create a staging route** with the `priority` strategy: the mock first, a real connection for the same model second ([routes](https://infro.io/docs/routes)). Set a per-attempt timeout, so the slow-headers drill has something to trip, and leave same-target retries at 0 so each fault costs one attempt.
3. **Issue a staging key** for the runner, pinned to the drill route, with a small spend limit ([limits](https://infro.io/docs/api/limits)). The standby is your own provider account and bills you for every request it serves, drills included.

Because routes and connections are environment-scoped with fail-closed matching, a staging key cannot reach production connections, and production keys never see the staging route.

The runner sets a fault, sends a request through the route, and then asserts against the request record rather than the response alone. The record names the chain position that served and lists every failed attempt with its error and latency ([traces](https://infro.io/docs/observability/traces)).

```python
import os
import time

import httpx

GATEWAY = "https://api.infro.io/v1"
H = {"Authorization": f"Bearer {os.environ['INFRO_DRILL_KEY']}"}   # staging key, pinned to the route
MOCK = os.environ["MOCK_URL"]                                       # https://drill-mock.example.com
CONTROL = {"Authorization": f"Bearer {os.environ['MOCK_CONTROL_TOKEN']}"}
ROUTE = "drill-support"
BODY = {"model": ROUTE, "max_tokens": 32,
        "messages": [{"role": "user", "content": "Reply with one short sentence."}]}


def set_fault(mode: str) -> None:
    httpx.post(f"{MOCK}/__fault", params={"mode": mode}, headers=CONTROL, timeout=10).raise_for_status()


def call() -> tuple[httpx.Response, str]:
    r = httpx.post(f"{GATEWAY}/chat/completions", json=BODY, headers=H, timeout=200)
    rid = r.headers.get("x-infro-request-id") or r.json().get("id")
    return r, rid


def record(rid: str) -> dict:
    for _ in range(15):                   # the record can take a moment to be written
        r = httpx.get(f"{GATEWAY}/requests/{rid}", headers=H, timeout=10)
        if r.status_code == 200:
            return r.json()
        time.sleep(1)
    raise AssertionError(f"no record for {rid}")


def expect_standby(r: httpx.Response, rec: dict, error: str | None = None) -> None:
    assert r.status_code == 200, f"user saw {r.status_code}"
    assert rec["route"] != "primary", "the faulty primary served"
    attempts = rec["fallback_attempts"]
    assert attempts and attempts[0]["route"] == "primary", "no failed primary attempt recorded"
    if error:
        assert attempts[0]["error"] == error, attempts[0]["error"]


def drill(mode: str, check) -> None:
    set_fault(mode)
    try:
        r, rid = call()
        check(r, record(rid))
        print(f"PASS {mode:22} {rid}")
    finally:
        set_fault("ok")


drill("503", lambda r, rec: expect_standby(r, rec, "upstream_error"))
drill("429", lambda r, rec: expect_standby(r, rec, "rate_limit_exceeded"))
drill("slow_headers", expect_standby)
drill("silent_after_headers", expect_standby)   # slow by design: the idle limit is at least 120 s
```

Two more drills check the connection-health rules INFRO documents for [providers](https://infro.io/docs/providers): five consecutive availability failures eject a connection, starting at a 30-second cool-off, while a 429 never counts.

```python
def primary_skipped(rec: dict) -> bool:
    return rec["route"] != "primary" and not rec["fallback_attempts"]


set_fault("ok")
call()                                    # one success resets the primary's failure count


# 1. Throttling never ejects: every request still tries the primary first.
set_fault("429")
try:
    for _ in range(7):
        r, rid = call()
        assert not primary_skipped(record(rid)), "a 429 ejected the primary"
finally:
    set_fault("ok")

# 2. Availability failures do: after five in a row the primary is skipped.
set_fault("503")
try:
    recs = [record(call()[1]) for _ in range(7)]
    assert primary_skipped(recs[-1]), "primary still in rotation after repeated 503s"
finally:
    set_fault("ok")

# 3. Recovery: after the cool-off, the primary serves again.
time.sleep(35)
assert record(call()[1])["route"] == "primary", "primary did not re-enter rotation"
print("PASS connection health")
```

Finish with the faults users can see. A mid-stream drop must reach the client as an error event with no `data: [DONE]`, which is the documented [streaming](https://infro.io/docs/api/streaming) behaviour once the first event has been sent; check that your interface marks or replaces the partial answer. And a second route whose only target is the mock covers the whole chain failing: in `429` mode the expected response is `429` with code `provider_rate_limited` and the mock's `Retry-After: 7`; in `503` mode it is `502 upstream_error` naming the connection tried, until repeated failures eject the mock and the route has nothing eligible left, at which point it becomes `503 no_available_provider` ([errors](https://infro.io/docs/api/errors)).

| Drill | Expected, from the documented behaviour | Evidence |
| --- | --- | --- |
| 503 on the primary | 200 from the standby | `route: standby_a`, attempt error `upstream_error` |
| 429 on the primary | 200 from the standby; primary never ejected | Attempt error `rate_limit_exceeded` on every request |
| Seven 503s in a row | Primary skipped after the fifth | Later records have no failed attempt |
| Slow headers | Standby after the per-attempt timeout | Attempt latency close to the timeout |
| Drop mid-stream | Error event, no `[DONE]` | Record with an error status on the serving connection |
| Only target down | 429 `provider_rate_limited` or 502 `upstream_error`; 503 `no_available_provider` once the mock is ejected | The message names the connection tried |

Table: Expected outcomes for the drill set. A drill passes only when the evidence matches, not when the request happens to succeed.

## Widening the blast radius, carefully

Staging drills prove the mechanism. They do not prove that production keys, quotas and deadlines line up. The next step is a small production experiment for faults that fail *before* a response starts: a `weighted` route that sends a small share of first attempts to the mock, which fails, so those requests fall through to the real targets. Users on that slice see only a later first token. Keep mid-stream faults out of production entirely, set a stop condition on the route's error rate, and remove the mock target when the experiment ends; the guidance is explicit that you must restore the workload to its known-good state afterwards[1].

Finally, drill the people as well as the code. Check that the `connection.failing` [alert](https://infro.io/docs/observability/alerts) for the ejected mock reached the channel your on-call engineer watches, and that the person who got it could find the request records from it. A failover that works silently while nobody knows the primary is down is only half a success.

## Checklist

- [ ] Write the steady state and hypothesis before injecting anything.
- [ ] Inject one fault at a time through a mock that speaks the real wire format.
- [ ] Cover errors before the response, slow headers, silence, a mid-stream drop and a full-chain failure.
- [ ] Assert on request records — chain position and failed attempts — not only on status codes.
- [ ] Confirm that throttling never ejects a target and that availability failures do.
- [ ] Confirm that an ejected target returns after its cool-off.
- [ ] Give the drill key a spend limit; the standby is a real, billed account.
- [ ] Run the automated drills on every routing change and on a schedule.
- [ ] Check that alerts fired and reached a person.

## Questions

### Should failover drills run in production?

Eventually, for faults that fail before a response starts, on a small share of traffic and with a stop condition. Start in a staging environment with the same route shape, and never inject mid-stream faults into production traffic, because users would see interrupted answers.

### How often should LLM failover drills run?

Run the automated set on every change to routing or client retry code, and on a fixed schedule such as weekly, since provider behaviour and your configuration drift without a code change. Keep a larger game day, with people and alerts involved, to a few times a year.

## Sources

1. REL12-BP04 Test resiliency using chaos engineering — AWS Well-Architected Framework. https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_testing_resiliency_failure_injection_resiliency.html (accessed 2026-09-29)
2. Principles of Chaos Engineering — principlesofchaos.org. https://principlesofchaos.org/ (accessed 2026-09-29)
3. Claude API errors — Anthropic. https://platform.claude.com/docs/en/api/errors (accessed 2026-09-29)
4. HTML Living Standard: Server-sent events — WHATWG. https://html.spec.whatwg.org/multipage/server-sent-events.html (accessed 2026-09-29)