Failover drills: test LLM provider failure before it happens

Failover code has an unusual property: it only runs when something else has already gone wrong. A retry loop, a fallback target, a stream-interruption handler — each sits unexecuted through months of healthy traffic, and then runs for the first time, at scale, during a provider incident. That is the worst possible moment to discover that the standby key was revoked, the timeout is longer than the client's deadline, or the error handler assumes every failure has a status code.

A drill runs that code on purpose, one fault at a time, while you are watching. The AWS Well-Architected reliability guidance lists "designing for resiliency, but not verifying" among the common anti-patterns it warns against, and recommends running fault-injection experiments regularly, as close to production as possible¹.

Treat each drill as an experiment

The Principles of Chaos Engineering describe an experiment in four steps: define steady state as a measurable output of normal behaviour, hypothesize that it continues under the fault, introduce a real-world fault, and try to disprove the hypothesis². That structure keeps a drill honest. Without a written hypothesis, any outcome can be read as success.

For a route serving a chat feature, steady state is a success rate, a p95 time to first token and a fallback share, taken from the request records before you start. A good hypothesis names the fault, the mechanism and the measurable outcome: if the primary returns 503, the standby serves every request within the client's 30-second deadline, the fallback share rises, and no user receives an error. The same principles ask you to minimize the blast radius and to automate experiments so they run continuously².

Choose faults from how providers actually fail

A drill is only as useful as the faults it injects. Provider failures cluster into a few shapes, and each exercises a different code path.

• Status errors before any response. A 429 with a wait hint, a 503, a 500, and on Anthropic's API a 529 overloaded_error³. These test classification: which errors move to another target, and which do not.
• Slow headers. The provider accepts the connection and takes a long time to answer. This tests per-attempt timeouts against the caller's deadline.
• Headers, then silence. The stream opens and nothing arrives. This tests idle limits, which are usually longer and less examined.
• A dropped stream. Some content arrives and the connection closes without a terminal event. A partial event at the cut is discarded by the SSE parser, as the specification requires⁴; the drill checks what your application does with the text that did arrive.
• Everything down. Every target fails. This tests the error your application finally receives, and whether it handles it.

Use your own incident history to add to the list; post-incident analyses are the guidance's recommended source for which faults to experiment with¹.

A mock upstream that fails on demand

The simplest reliable fault injector is a small server that speaks the provider's wire format and fails in whichever way it is told. It exercises your real client and gateway code, including parsing, instead of a mocked function. The server below implements the OpenAI-compatible chat completions endpoint, streaming and non-streaming, with a control endpoint to switch modes.

import { createServer, type ServerResponse } from "node:http";

type Mode = "ok" | "429" | "503" | "slow_headers" | "silent_after_headers"
  | "drop_mid_stream" | "error_mid_stream";

let mode: Mode = (process.env.FAULT_MODE as Mode | undefined) ?? "ok";
const CONTROL_TOKEN = process.env.CONTROL_TOKEN ?? "";   // protects the mode switch
const UPSTREAM_KEY = process.env.UPSTREAM_KEY ?? "";     // the key the gateway sends
const CUT_AFTER = Number(process.env.CUT_AFTER ?? 3);
const SLOW_MS = Number(process.env.SLOW_MS ?? 30_000);

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
const now = () => Math.floor(Date.now() / 1000);

function send(res: ServerResponse, payload: unknown) {
  res.write(`data: ${JSON.stringify(payload)}\n\n`);
}

function chunk(delta: Record<string, string>, finish: string | null = null) {
  return {
    id: "chatcmpl-drill", object: "chat.completion.chunk", created: now(), model: "drill-model",
    choices: [{ index: 0, delta, finish_reason: finish }],
  };
}

function json(res: ServerResponse, status: number, body: unknown, headers: Record<string, string> = {}) {
  res.writeHead(status, { "content-type": "application/json", ...headers });
  res.end(JSON.stringify(body));
}

createServer(async (req, res) => {
  const url = new URL(req.url ?? "/", "http://drill.local");

  if (req.method === "POST" && url.pathname === "/__fault") {
    if (!CONTROL_TOKEN || req.headers.authorization !== `Bearer ${CONTROL_TOKEN}`) return json(res, 401, {});
    mode = (url.searchParams.get("mode") as Mode | null) ?? "ok";
    return json(res, 200, { mode });
  }
  if (req.method !== "POST" || !url.pathname.endsWith("/chat/completions")) return json(res, 404, {});
  if (UPSTREAM_KEY && req.headers.authorization !== `Bearer ${UPSTREAM_KEY}`) {
    return json(res, 401, { error: { message: "bad key", type: "invalid_request_error" } });
  }

  let raw = "";
  for await (const part of req) raw += part;
  const body = JSON.parse(raw || "{}") as { stream?: boolean };
  const words = ["Drill", " answer", " from", " the", " mock", " upstream."];

  if (mode === "429") {
    return json(res, 429, { error: { message: "Rate limit reached (drill)", type: "requests" } }, { "retry-after": "7" });
  }
  if (mode === "503") {
    return json(res, 503, { error: { message: "Temporarily overloaded (drill)", type: "server_error" } });
  }
  if (mode === "slow_headers") await sleep(SLOW_MS);         // then answer normally
  if (mode === "silent_after_headers") {
    res.writeHead(200, { "content-type": "text/event-stream" });
    res.flushHeaders();
    return;                                                   // never writes, never ends
  }

  if (!body.stream) {
    return json(res, 200, {
      id: "chatcmpl-drill", object: "chat.completion", created: now(), model: "drill-model",
      choices: [{ index: 0, message: { role: "assistant", content: words.join("") }, finish_reason: "stop" }],
      usage: { prompt_tokens: 12, completion_tokens: 6, total_tokens: 18 },
    });
  }

  res.writeHead(200, { "content-type": "text/event-stream", "cache-control": "no-cache" });
  send(res, chunk({ role: "assistant", content: "" }));
  for (const [i, word] of words.entries()) {
    if (i === CUT_AFTER && mode === "drop_mid_stream") return void res.destroy();   // no terminator
    if (i === CUT_AFTER && mode === "error_mid_stream") {
      send(res, { error: { message: "Stream failed (drill)", type: "server_error" } });
      return void res.end();
    }
    send(res, chunk({ content: word }));
    await sleep(150);
  }
  send(res, chunk({}, "stop"));
  res.write("data: [DONE]\n\n");
  res.end();
}).listen(Number(process.env.PORT ?? 8787));

Point a client or SDK straight at it to test your own retry and stream handling in isolation. To drill a gateway, put it behind HTTPS and register it as a provider target, as below.

Drilling a route in INFRO

INFRO lets you add the mock as an ordinary connection, so drills run through the same routing, failover and records as production traffic, without touching a production route.

1. Connect the mock as an OpenAI-compatible endpoint: its HTTPS base URL, the key it expects, and a model map from the catalog model your route serves to drill-model. A model with no map entry is not served through that kind of connection. Scope the connection to the staging environment.
2. Create a staging route with the priority strategy: the mock first, a real connection for the same model second (routes). Set a per-attempt timeout, so the slow-headers drill has something to trip, and leave same-target retries at 0 so each fault costs one attempt.
3. Issue a staging key for the runner, pinned to the drill route, with a small spend limit (limits). The standby is your own provider account and bills you for every request it serves, drills included.

Because routes and connections are environment-scoped with fail-closed matching, a staging key cannot reach production connections, and production keys never see the staging route.

The runner sets a fault, sends a request through the route, and then asserts against the request record rather than the response alone. The record names the chain position that served and lists every failed attempt with its error and latency (traces).

import os
import time

import httpx

GATEWAY = "https://api.infro.io/v1"
H = {"Authorization": f"Bearer {os.environ['INFRO_DRILL_KEY']}"}   # staging key, pinned to the route
MOCK = os.environ["MOCK_URL"]                                       # https://drill-mock.example.com
CONTROL = {"Authorization": f"Bearer {os.environ['MOCK_CONTROL_TOKEN']}"}
ROUTE = "drill-support"
BODY = {"model": ROUTE, "max_tokens": 32,
        "messages": [{"role": "user", "content": "Reply with one short sentence."}]}


def set_fault(mode: str) -> None:
    httpx.post(f"{MOCK}/__fault", params={"mode": mode}, headers=CONTROL, timeout=10).raise_for_status()


def call() -> tuple[httpx.Response, str]:
    r = httpx.post(f"{GATEWAY}/chat/completions", json=BODY, headers=H, timeout=200)
    rid = r.headers.get("x-infro-request-id") or r.json().get("id")
    return r, rid


def record(rid: str) -> dict:
    for _ in range(15):                   # the record can take a moment to be written
        r = httpx.get(f"{GATEWAY}/requests/{rid}", headers=H, timeout=10)
        if r.status_code == 200:
            return r.json()
        time.sleep(1)
    raise AssertionError(f"no record for {rid}")


def expect_standby(r: httpx.Response, rec: dict, error: str | None = None) -> None:
    assert r.status_code == 200, f"user saw {r.status_code}"
    assert rec["route"] != "primary", "the faulty primary served"
    attempts = rec["fallback_attempts"]
    assert attempts and attempts[0]["route"] == "primary", "no failed primary attempt recorded"
    if error:
        assert attempts[0]["error"] == error, attempts[0]["error"]


def drill(mode: str, check) -> None:
    set_fault(mode)
    try:
        r, rid = call()
        check(r, record(rid))
        print(f"PASS {mode:22} {rid}")
    finally:
        set_fault("ok")


drill("503", lambda r, rec: expect_standby(r, rec, "upstream_error"))
drill("429", lambda r, rec: expect_standby(r, rec, "rate_limit_exceeded"))
drill("slow_headers", expect_standby)
drill("silent_after_headers", expect_standby)   # slow by design: the idle limit is at least 120 s

Two more drills check the connection-health rules INFRO documents for providers: five consecutive availability failures eject a connection, starting at a 30-second cool-off, while a 429 never counts.

def primary_skipped(rec: dict) -> bool:
    return rec["route"] != "primary" and not rec["fallback_attempts"]


set_fault("ok")
call()                                    # one success resets the primary's failure count


# 1. Throttling never ejects: every request still tries the primary first.
set_fault("429")
try:
    for _ in range(7):
        r, rid = call()
        assert not primary_skipped(record(rid)), "a 429 ejected the primary"
finally:
    set_fault("ok")

# 2. Availability failures do: after five in a row the primary is skipped.
set_fault("503")
try:
    recs = [record(call()[1]) for _ in range(7)]
    assert primary_skipped(recs[-1]), "primary still in rotation after repeated 503s"
finally:
    set_fault("ok")

# 3. Recovery: after the cool-off, the primary serves again.
time.sleep(35)
assert record(call()[1])["route"] == "primary", "primary did not re-enter rotation"
print("PASS connection health")

Finish with the faults users can see. A mid-stream drop must reach the client as an error event with no data: [DONE], which is the documented streaming behaviour once the first event has been sent; check that your interface marks or replaces the partial answer. And a second route whose only target is the mock covers the whole chain failing: in 429 mode the expected response is 429 with code provider_rate_limited and the mock's Retry-After: 7; in 503 mode it is 502 upstream_error naming the connection tried, until repeated failures eject the mock and the route has nothing eligible left, at which point it becomes 503 no_available_provider (errors).

Widening the blast radius, carefully

Staging drills prove the mechanism. They do not prove that production keys, quotas and deadlines line up. The next step is a small production experiment for faults that fail before a response starts: a weighted route that sends a small share of first attempts to the mock, which fails, so those requests fall through to the real targets. Users on that slice see only a later first token. Keep mid-stream faults out of production entirely, set a stop condition on the route's error rate, and remove the mock target when the experiment ends; the guidance is explicit that you must restore the workload to its known-good state afterwards¹.

Finally, drill the people as well as the code. Check that the connection.failing alert for the ejected mock reached the channel your on-call engineer watches, and that the person who got it could find the request records from it. A failover that works silently while nobody knows the primary is down is only half a success.

Checklist

• Write the steady state and hypothesis before injecting anything.
• Inject one fault at a time through a mock that speaks the real wire format.
• Cover errors before the response, slow headers, silence, a mid-stream drop and a full-chain failure.
• Assert on request records — chain position and failed attempts — not only on status codes.
• Confirm that throttling never ejects a target and that availability failures do.
• Confirm that an ejected target returns after its cool-off.
• Give the drill key a spend limit; the standby is a real, billed account.
• Run the automated drills on every routing change and on a schedule.
• Check that alerts fired and reached a person.