Is your fallback model good enough? Test fallback equivalence

Why a configured fallback proves nothing yet

Configuring failover answers one question: when the primary fails, does a request get served by something else? It does not answer whether that something else does the job. A fallback that returns HTTP 200 with a malformed object, a skipped tool call or a truncated answer is worse than an error, because nothing alerts on it.

The mechanics make this more likely than it looks. INFRO's model fallbacks are an opt-in fallbacks array of up to five catalog model IDs, tried in order once every connection for the primary model has failed before the first token.¹ The trigger is broad: a 5xx, a timeout, a 429, a rejected credential, a 404, or even a 400/422, because providers differ on which optional parameters they accept.¹ So the fallback also receives requests the primary rejected, which are often your strangest ones: unusually large prompts, odd tool definitions, unsupported parameters.

This post gives a small eval you run once per primary-fallback pair, and again when anything in the pair changes. It measures three things separately: compatibility, quality gap and cost per successful task. For how to rehearse the failure itself, see failover drills. For routing on quality at normal times, see quality-aware routing.

Define the pair and the contract first

Write down, per pair, what the fallback must do. Without that you will grade it on vibes after the fact.

• Pair: primary model id, fallback model id, and whether they share a vendor. Pin snapshots where the vendor offers them, otherwise the pair you tested may not be the pair that runs.
• Task types: the distinct jobs behind this route (extraction, classification, drafting, agent step). Fallbacks are rarely tested per task, and quality gaps are rarely uniform across tasks.
• Output contract: the JSON schema or the free-text rubric each task is held to.
• Tool contract: the exact tools array the application sends, including tool names, parameter schemas and tool_choice.
• Allowed gap: the largest pass-rate drop you will accept on each task type, chosen before you see results. For example, a drop of up to 3 points on drafting, zero tolerance on schema validity.
• Status of the fallback model: a stable model, not a preview. Google's models page labels models Stable or Preview, and some as legacy with a recommended replacement.² A fallback that sits on a preview or legacy id can disappear on a schedule you do not control.

Candidate pairs are often adjacent tiers: a flagship primary and a small or previous-generation fallback. Anthropic's newsroom, for instance, lists Sonnet 5.5 (announced Sep 28, 2026) and Haiku 5.5 (Oct 7, 2026, described as its fastest small model).³ Whether such a pair is acceptable is exactly what the method below decides; the announcement does not.

Build the case set from your traffic

Sample real requests, stratified by task type, tool use, schema and prompt size. A hand-written set of twenty friendly prompts will pass everything.

1. Pull a sample per task type. Aim for 100 to 300 cases per type that matters. Include the long tail: the biggest prompts, the longest tool lists, multilingual inputs.
2. Oversample failure-shaped traffic. Include requests that previously produced 400s, truncations or retries, since those are what lands on the fallback.
3. Respect your logging policy. Prompt and completion logging in INFRO traces is off by default with a retention window you choose, so a sample may need to come from your own application logs or from redacted or synthetic inputs. See the payload logging policy before copying production text into an eval set.
4. Freeze expected behavior. For extraction, store the reference object. For tools, store the expected tool name and the argument constraints. For open-ended tasks, store the rubric.
5. Version the set. Cases, schemas, tool definitions and rubric live in one repository so a run is reproducible.

Name the model directly in each request rather than relying on a real outage to exercise the fallback. Naming a model sends the request to that model over your own connections, and the response reports the model, provider and connection that served it, which you should log with each result.¹

Gate 1: schema and tool-call compatibility

Compatibility is pass/fail and cheap to automate, so run it first. A fallback that fails here does not need a quality comparison.

Schema checks

For each case with a response_format, run both models and validate the output with a real JSON Schema validator. Do not eyeball it. Record four outcomes: parse failure, schema violation, valid, and valid-but-wrong (caught in the quality gate).

Three known traps:

• Enforcement differs by path. json_schema with strict: true only works on models whose vendor supports it, and INFRO does not record per-model support yet, so you check vendor documentation and test.⁴
• Anthropic connections drop response_format. INFRO drops it when a request is served by an Anthropic connection, so the reply is ordinary text; the docs suggest describing the shape in the prompt or using a single tool whose parameters are your schema.⁴ If a fallback chain crosses into an Anthropic model, the primary's decoding-time guarantee silently becomes a prompt-time hope, and your schema pass rate is the only evidence you have.
• Strict-mode schema rules. Strict mode wants additionalProperties: false on every object and every property listed in required, with optional fields as nullable types.⁴ A schema that works on the primary may be rejected by the fallback's provider with a 400 carrying that provider's message.

Vendors expose structured output in their own native shape too; Gemini's API takes a JSON Schema and returns responses that adhere to it,⁵ but the request field names differ from OpenAI-style response_format. Test through the same endpoint your application uses, not the vendor playground.

Tool-call checks

Tools are where cross-vendor fallbacks break quietly. INFRO uses one OpenAI-style tools and tool_choice schema, forwards it as is to providers that speak that protocol, and translates it to and from Anthropic's tool_use blocks for Anthropic connections.⁶ It also notes that it does not record tool support per model, that capability sets differ between a vendor's flagship and its small models, and that you should test at integration time.⁶ Gemini's own function calling lets the model decide when to call a function and supply parameters,⁷ but whether it picks the right function with the right arguments for your tool list is an empirical question.

Per case, check:

• the model called a tool when it should have, and not when it should not;
• the tool name is one you defined;
• function.arguments parses as JSON (it arrives as a string and is model-generated)⁶ and validates against that tool's parameter schema;
• required arguments are present and enum values are legal;
• parallel calls behave as the application expects, and the loop terminates when tool results are appended with matching tool_call_ids.

For agent routes, run the full loop with stubbed tool results, three to five steps deep. A model that makes a valid first call and drifts at step four only shows up here.

Gate 2: the quality gap

Compatibility says the output is well-formed. Quality says it is right. Compare the fallback against the primary on the same inputs, as pairs.

For tasks with a reference answer (extraction, classification, routing decisions), score exact or field-level match. For open-ended tasks, use a rubric grader. If that grader is an LLM, calibrate it against human labels first; this post covers how. Keep the grader a different model from either member of the pair.

Report, per task type:

• pass rate for each model, with an interval;
• the discordant pairs: cases the primary passed and the fallback failed, and the reverse;
• the five worst fallback failures, read by a person.

The discordant count is the useful number. Two models can both pass 94% and still fail on different cases. Run each case two or three times if you cannot fix sampling parameters, so run-to-run variance does not get blamed on the model.

If the pair fails, you have options short of abandoning it: tighten the prompt for the fallback, add a repair retry, route only some task types to that fallback, or choose a different fallback. Each change restarts the eval for that pair.

Gate 3: cost per successful task

A fallback changes the bill in two ways: the rate differs, and the number of tokens differs, because tokenizers, verbosity and retries differ. Rates alone mislead. Compute cost per successful task, the ratio of total spend to tasks that passed gates 1 and 2; see cost per successful task for the general metric.

From each response take the token counts, price them with the rates your provider actually bills you (negotiated rates where you have them, list rates otherwise), and divide by the number of passes. INFRO records estimated provider cost per request and lets you enter contract rates that estimates use before list rates; the figures are estimates of what your provider will bill, and the provider invoice remains the truth.⁸

import json, os
from collections import defaultdict
from jsonschema import validate, ValidationError
from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1",
                api_key=os.environ["INFRO_API_KEY"])

# Placeholder rates in USD per million tokens. Replace with what
# your provider bills you.
RATES = {"primary-model-id": (3.00, 15.00), "fallback-model-id": (1.00, 5.00)}

def check_case(model, case):
    kwargs = {"model": model, "messages": case["messages"]}
    if case.get("tools"):
        kwargs["tools"] = case["tools"]
    if case.get("response_format"):
        kwargs["response_format"] = case["response_format"]
    r = client.chat.completions.create(**kwargs)
    msg = r.choices[0].message
    ok, why = True, ""
    if case.get("tools") and case["expect_tool"]:
        calls = msg.tool_calls or []
        if not calls:
            ok, why = False, "no_tool_call"
        else:
            spec = {t["function"]["name"]: t["function"]["parameters"]
                    for t in case["tools"]}
            try:
                for c in calls:
                    args = json.loads(c.function.arguments)
                    validate(args, spec[c.function.name])
            except (KeyError, json.JSONDecodeError, ValidationError) as e:
                ok, why = False, type(e).__name__
    elif case.get("schema"):
        try:
            validate(json.loads(msg.content or ""), case["schema"])
        except (json.JSONDecodeError, ValidationError) as e:
            ok, why = False, type(e).__name__
    pin, pout = RATES[model]
    cost = (r.usage.prompt_tokens * pin + r.usage.completion_tokens * pout) / 1e6
    return {"ok": ok, "why": why, "cost": cost, "served": r.model}

def run_pair(cases, primary, fallback):
    out = defaultdict(list)
    for case in cases:
        for m in (primary, fallback):
            out[m].append(check_case(m, case))
    for m, rows in out.items():
        passed = sum(r["ok"] for r in rows)
        spend = sum(r["cost"] for r in rows)
        print(m, f"pass={passed}/{len(rows)}",
              f"cost_per_success={spend / max(passed, 1):.5f}")
    return out

The script covers gate 1 and the cost half of gate 3. Add your quality scorer for gate 2 and merge its verdict into ok so cost per success counts only answers that are actually right. Log served for each row, so you can prove the fallback model answered what you thought it did.

Cost matters less for a fallback than for a primary, since it carries traffic only during incidents. Multiply the per-success cost delta by the traffic you expect it to carry in a bad hour, and check that figure against your budgets. Tool-heavy and long-context traffic can move the number far more than average traffic does.

Decide, record and keep it current

Turn the three gates into a verdict you can read in a pull request.

Store the verdict beside the route config: pair, date, case-set version, pass rates and intervals, cost per success, and the task types the fallback is not approved for. If a task type is not approved, give it a route that fails with an error rather than one that fails over, since an honest error is better than a plausible wrong answer.

Re-run when a trigger fires:

• the primary or fallback is repinned, or a vendor moves an alias to a new snapshot (see aliases vs pinned snapshots);
• a schema or a tool definition changes;
• a vendor announces a retirement for either model;
• the share of traffic landing on the fallback changes.

In production, INFRO can help with the monitoring half of this. Failover retries elsewhere without the client seeing it, and the response names what served the request, so fallback traffic is visible per request.¹ Eval monitors can score a share of served chat answers against your rubric in the background and summarize per served model, so you get a running quality number for the fallback when it serves. INFRO sees request metadata rather than judging answers itself, so the rubric and the decision model are yours. None of this replaces the offline pair eval above, which is what you run before the fallback is needed.