Failover mid-agent-run: what happens to tool calls and state

What failover does, and does not do, to a run

Failover is a per-request mechanism. In INFRO's documented behavior, connection failover tries the same model on another of your connections, and model fallbacks try an ordered list of other models, and both operate strictly before the first token.¹ If the failing provider returns a 5xx, times out or answers 429 before the response starts, the next target is tried and your client sees only a higher time to first token.¹

An agent run is a different shape: a loop of model requests with your code executing tools between them. That gives three consequences worth stating plainly:

• A tool call is never "in flight" at the provider. The model emits a request to call a tool, and your application runs it. Anthropic's docs draw the same line: client tools run in your application, and your code sends back a tool_result.²
• Failover decides who answers the next request. Whatever that request carries (the whole history, tool definitions, prior tool results) is now read by a different model, possibly through a different message format.
• The things that actually break are outside the request: side effects that ran twice, state that lived only in memory, and a prompt cache that no longer applies.

So the question to ask is not "does the tool survive failover" but "what does the run look like if any single model request is answered by a different provider than the one before it, or not answered at all".

Five failure modes, ranked by damage

The first one is the only one that costs money or trust outside your own systems, so it gets the most attention below. The rest are correctness and cost problems you can test for.

Checkpoint every step, in your own format

Do not let the provider's response object be your run state. Write a journal that is append-only and survives a worker restart. Two writes per step are enough:

1. Before executing any tool: persist the assistant message exactly as received, with its tool calls.
2. After each tool finishes: persist that tool's result, keyed by the call's position in the step.

The "exactly as received" rule matters for a concrete reason. INFRO's tool-calling docs say to append the assistant message exactly as received because it carries the tool_calls your results answer, and to answer each call with a tool message carrying the matching tool_call_id.³ Anthropic is stricter about ordering: tool results must immediately follow their tool use blocks, and the tool_result blocks must come first in the user message.⁴ A journal that stores the pair intact lets you rebuild a valid history after a crash.

from dataclasses import dataclass, field

@dataclass
class Step:
    index: int
    assistant: dict                      # message as received, untouched
    results: dict[int, str] = field(default_factory=dict)  # call position -> result

def resume(run_id: str, store) -> list[dict]:
    """Rebuild neutral history; return pending call positions for the last step."""
    steps = store.load_steps(run_id)
    history = []
    for s in steps:
        history.append(s.assistant)
        calls = s.assistant.get("tool_calls", [])
        for pos, call in enumerate(calls):
            if pos in s.results:
                history.append({"role": "tool",
                                "tool_call_id": call["id"],
                                "content": s.results[pos]})
    return history

On resume, find the last step whose assistant message has calls without stored results. Those are the only calls you may need to execute, and each must go through the idempotency check described next. Never re-ask the model for a step whose assistant message is already in the journal; replaying the stored message is both free and deterministic.

Idempotency keys for side-effecting tools

The model's tool call id is the wrong key. If a run is retried from the previous request, a new model answer carries a new id, and sometimes different arguments. Derive the key from things your journal controls: run id, step index and the call's position within the step.

import hashlib, json

def effect_key(run_id: str, step: int, pos: int) -> str:
    return f"{run_id}:{step}:{pos}"

def args_hash(args: dict) -> str:
    canon = json.dumps(args, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canon.encode()).hexdigest()

def run_effect(db, run_id, step, pos, tool, args, do):
    key, h = effect_key(run_id, step, pos), args_hash(args)
    row = db.get_effect(key)
    if row is None:
        db.insert_effect(key, tool, h, status="started")   # unique on key
        result = do(args, idempotency_key=key)
        db.finish_effect(key, result)
        return result
    if row.args_hash != h:
        return "error: same step re-planned with different arguments; needs review"
    if row.status == "finished":
        return row.result                                  # replay, do not execute
    return do(args, idempotency_key=key)                   # crashed mid-flight: retry with same key

Three details carry the design. The unique insert makes two workers racing on the same step safe. The arguments hash turns "the model re-planned differently after the switch" into an explicit refusal instead of a second action. And the final branch, a crash between starting and finishing, only works if the downstream system honors the key. For your own services, enforce it with a unique constraint. For third-party APIs, use their idempotency mechanism where one exists; where none exists, mark the tool non-retryable and surface the ambiguity to a human rather than guessing.

Tools exposed over MCP deserve the same treatment. The MCP docs describe tools as functions the model decides to call, which can write to databases, call external APIs and modify files.⁵ Nothing in that description makes a call idempotent, so the key has to travel in the tool's own arguments or be enforced by the server.

Where mid-stream failures fall outside failover

Because failover happens only before the first token, a stream that dies halfway is your agent's problem. INFRO's streaming docs state that after a mid-stream error event the connection closes without the [DONE] sentinel, so connection close must be treated as terminal.⁶ For an agent that streams tool-call arguments, that leaves a partial arguments string.

The safe handling is short:

• Discard the partial assistant message. Do not parse half a JSON argument string and execute it. The tool-calling docs already warn that function.arguments is model-generated JSON in a string, to be parsed defensively and validated before anything with side effects runs.³
• Journal nothing for that step, then re-request it. Because no tool ran, replay is safe.
• Count the re-request against the run's step and retry budget, so an unstable provider cannot loop the agent. The loop-bounding article covers those limits.

The dangerous variant is a stream that completes with tool calls, your code runs them, and the process dies before the result is journaled. That is the case the effect table above exists for.

Message-format translation hazards

When the next request lands on another provider, the history is translated. INFRO's docs say that for Anthropic connections it translates tools, tool_choice, tool calls and tool results to and from Anthropic's tool_use blocks.³ That covers the common shapes. It does not make provider-specific extras portable. Check these in your own history:

• Thinking blocks. Claude's thinking blocks carry a signature, an encrypted copy of the reasoning that is passed back unchanged in multi-turn and tool-use conversations.⁷ Treat it as meaningful only to the provider that issued it. When history crosses providers, keep a plain-text summary in your journal if you want continuity, and do not assume another provider accepts the block.
• Tool call ids. Ids are opaque and provider-issued. Always pair results to calls by position in your journal, then render the id the destination expects.
• Parallel calls. One assistant turn may hold several calls. Claude expects all matching tool_result blocks in one user message, first in the content array;⁴ OpenAI-style history uses one tool message per call.³ Your renderer must emit the right grouping.
• Tool result content. Claude accepts nested text, image and document blocks in a result.⁴ An OpenAI-style tool message takes a string. A result containing an image needs a fallback representation on the string side.
• System content and caching markers. Anthropic cache breakpoints are explicit cache_control markers, while OpenAI caching is automatic.⁸ Neutral history should not store markers; add them at render time.

The pattern that holds up: store neutral messages (role, text, calls, results, attachments), and write one renderer per provider family. A switch is then a re-render of the same journal, not a repair job. Whether the fallback model is competent on your tool schemas is a separate question; the fallback equivalence test covers it.

The cold cache after a switch

Provider-side caches belong to the upstream that served the request. INFRO's docs say hits depend on consecutive requests reaching the same connection, and that a failover to another connection resets the cache for that prefix, which is then billed as a cold prefix.⁸ For an agent this compounds, because every step re-sends the whole growing history. The step after a switch pays full input price on the entire transcript, and a long run pays it again if traffic flips back.

Two things limit the damage. Keep the prefix stable and append-only (system prompt, then tools in a fixed order, then history), which the docs recommend for cache hits in general.⁸ And do not flap: a route that fails over and fails back on every health blip pays the cold cost repeatedly. The failover bill article works through the cost side.

What the gateway can show you, and what stays yours

On INFRO, failover and health-aware routing work on a named route over your own connections, and apply to chat completions including streaming and tool calling. Each request's trace records which connection served it, so you can see in one place that step 19 ran on a different provider than step 18. INFRO executes no tool, so idempotency and the journal stay in your application. If your tools are MCP servers registered through the MCP gateway, their calls land in a call log on the same trace as the model calls.

A test plan before you ship

• Kill the worker after a write tool returns but before its result is journaled, then resume. Exactly one external effect should exist.
• Cut a stream after half of a tool call's arguments. Nothing should execute, and the step should be re-requested.
• Force a provider switch at every step index of a recorded run, one at a time. History must render validly for each provider.
• Re-plan a step with different arguments on resume. The effect layer must refuse and flag it.
• Include a run with parallel tool calls and one with an image in a tool result across the switch.
• Compare per-step input cost around a switch against steady state, and confirm the spike matches your cold-cache estimate.

The failover drills article describes how to stage provider failure; run these cases inside a drill, with real tools replaced by recording stubs.