Same open-weight model, different hosts: a consistency test

Open-weight models are attractive partly because many hosts serve them. The same model id on four inference providers looks like four interchangeable suppliers: pick on rate or latency, fail over between them, move when one has a bad week. That is true only if the four behave the same, and nothing guarantees they do.

This article is a test design. It describes what can differ between hosts, a protocol for measuring it, the harness to run it, and how to interpret a difference. It publishes no results, because we did not run it for this article, and any host comparison you read should come with the method and noise band behind it.

What differs when two hosts serve the same weights

Precision. Hosts routinely serve reduced-precision weights. Together describes quantizing to FP8, FP4 and hybrid precision with calibration and selective mixed precision on sensitive paths, and calls the result near-lossless — its own claim¹. Fireworks notes that a deployment uses the FP16 checkpoint by default, that each model lists the precisions it has been prepared for, and that quantization can change model numerics and introduce small changes to the output². Groq serves models with its TruePoint numerics, which it says reduce precision only where accuracy is not affected³. DeepInfra prints fp8 or fp4 on each model listing⁴. Some models also ship quantized by their authors: gpt-oss-120b's MoE weights are MXFP4 in the release itself⁵.

Chat template. Messages, tool definitions and reasoning settings must be rendered into tokens the way the model was trained. The gpt-oss card says the models should only be used with the harmony response format and will not work correctly otherwise⁵. A host with a subtly different template produces subtly different behaviour, most visibly in tool calls.

Sampler support and defaults. Hosts differ in which parameters they honour, their defaults when you send none, and whether seed does anything.

Limits. A host can serve a shorter context window or lower maximum output than the model card allows. Groq, for example, lists its own context window and maximum output for each model it serves³.

Features. Tool-call parsing and JSON schema enforcement are host software, not weights. One host may support schema-constrained decoding for a model that another serves only in JSON-object mode.

Runtime nondeterminism. Even one host is not deterministic. vLLM states it does not guarantee reproducible results by default, for performance, and only provides reproducibility on the same hardware and version⁶. Thinking Machines showed the main cause is kernels that are not batch-invariant, so a request's output depends on server load: 1,000 temperature-0 completions of one prompt produced 80 distinct outputs, all identical for the first 102 tokens⁷.

That last point is why most informal host comparisons mislead. If two runs on the same host already differ, a single run on each of two hosts shows noise, not a host difference.

What hosts publish about precision

Record these statements, with the date, before you test. They tell you where to look first, and they change: a host can requantize a model without changing its name.

The protocol

1. Pin the model. One model card, one revision where hosts expose it, and identical request parameters everywhere: temperature 0, the same max_tokens, the same system prompt, no host-specific extras.
2. Record each host's stated configuration: precision, context window, maximum output, tool and JSON support. If a host does not state precision, record that.
3. Measure noise first. Send every prompt five to ten times to one host, spread over time. Compute, per prompt, how often the repeats agree and how the checkable score varies. This is your noise band.
4. Run the suite on every host, interleaving hosts in time so a bad hour at one does not look like a quality gap, with the same number of repeats.
5. Score with checks, not opinions. Use tasks with a verifiable answer: extraction with a known result, arithmetic, classification with labels, JSON that must validate, tool calls that must name the right function with parseable arguments. Also record finish_reason, output tokens and latency.
6. Report with the method. Dates, host settings, sample sizes, the noise band and the raw outputs.

Fireworks makes a related point about evaluating quantization: task accuracy scores correctness as a step function, so small shifts in the token distribution are hidden by benchmark noise, and it measures divergence in token probabilities instead⁸. You usually cannot read logits from a hosted API, which is another reason to use many repeats and checkable tasks rather than a single benchmark number.

The harness

The harness below sends each prompt to several targets, repeats it, checks each answer, and writes one JSON line per response. It uses the OpenAI SDK, so it works against any OpenAI-compatible endpoint; here each target is an INFRO route pinned to one host, described in the next section.

import json
import os
import time
from collections import defaultdict

from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1", api_key=os.environ["INFRO_API_KEY"])

# One route per host, each with a single target: the same model on one connection.
TARGETS = ["qwen-host-a", "qwen-host-b", "qwen-host-c"]
REPEATS = 8
RUN_ID = time.strftime("%Y%m%dT%H%M%S")

# Every case has a checker, so scoring needs no judgement.
CASES = [
    {"id": "sum-1", "prompt": "What is 1847 + 2395? Reply with the number only.",
     "check": lambda text: text.strip() == "4242"},
    {"id": "label-1", "prompt": "Classify the sentiment as positive, negative or neutral. "
     "Reply with one word.\n\nThe refund arrived a week late and nobody answered my emails.",
     "check": lambda text: text.strip().lower() == "negative"},
    {"id": "json-1", "prompt": 'Return JSON only: {"city": string, "year": integer} for '
     '"The treaty was signed in Lisbon in 2007."',
     "check": lambda text: _json_equals(text, {"city": "Lisbon", "year": 2007})},
]


def _json_equals(text: str, expected: dict) -> bool:
    try:
        return json.loads(text) == expected
    except json.JSONDecodeError:
        return False


def run_once(target: str, case: dict, repeat: int) -> dict:
    started = time.monotonic()
    completion = client.chat.completions.create(
        model=target,
        messages=[{"role": "user", "content": case["prompt"]}],
        temperature=0,
        max_tokens=256,
        extra_body={"metadata": {"tags": {"run": RUN_ID, "case": case["id"]}}},
    )
    body = completion.model_dump()
    text = completion.choices[0].message.content or ""
    return {
        "run": RUN_ID, "target": target, "case": case["id"], "repeat": repeat,
        "provider": body.get("provider"), "connection": body.get("connection"),
        "request_id": completion.id, "finish_reason": completion.choices[0].finish_reason,
        "output_tokens": completion.usage.completion_tokens if completion.usage else None,
        "latency_s": round(time.monotonic() - started, 3),
        "passed": case["check"](text), "text": text,
    }


def main() -> None:
    rows = []
    with open(f"consistency-{RUN_ID}.jsonl", "w") as out:
        for repeat in range(REPEATS):
            for case in CASES:
                for target in TARGETS:  # interleave hosts in time
                    row = run_once(target, case, repeat)
                    rows.append(row)
                    out.write(json.dumps(row) + "\n")

    by_target = defaultdict(list)
    agreement = defaultdict(set)
    for row in rows:
        by_target[row["target"]].append(row["passed"])
        agreement[(row["target"], row["case"])].add(row["text"])

    for target, results in by_target.items():
        distinct = [len(agreement[(target, c["id"])]) for c in CASES]
        print(f"{target}: pass {sum(results)}/{len(results)}, "
              f"distinct outputs per case {distinct}")


if __name__ == "__main__":
    main()

Three cases are a sketch; a real suite has a hundred or more drawn from production traffic, grouped by task type so a failure can be traced to a capability. Add cases for tool calls with a checker that parses the arguments, and long-context cases that sit near each host's stated limit.

Reading the results

Look for concentration before concluding anything. Failures that cluster in one task type point at a feature or limit difference you can often fix. Failures spread thinly across every task type, beyond the noise band, point at the weights or the template, and are a reason to keep that workload on the hosts that pass. Rerun the suite whenever a host announces an engine change, and on a schedule, because hosts change precision and runtime without changing model names.

Running the test through INFRO

In INFRO each host is a provider connection on your own account — Together, Fireworks and Groq are connection kinds, and any OpenAI-compatible host can be added as an OpenAI-compatible endpoint. Together and Fireworks model ids carry host-specific prefixes, so each model is named explicitly in the connection's model map. See providers.

To pin a request to one host, create one route per host, each with a single target: the model and the connection that should serve it. The harness then calls qwen-host-a, qwen-host-b and so on. Every response names the provider and connection that served it, and route reads primary when no failover happened — check it, because a failover inside a single-target route cannot happen, but a misconfigured route with two targets would silently mix hosts.

The metadata.tags in the harness are recorded on each request and never sent to the provider, so the run can be found again in the request log with GET /v1/requests, or exported as CSV with GET /v1/requests/export, including latency, token counts and estimated provider cost per request. See request traces. Estimated provider cost is what your host will bill you, at its list rate or the rate you entered; INFRO's own subscription is separate and does not vary with the host.

Once the test is done, put only the hosts that passed on the production route, in priority order or by weight. INFRO's traffic changes also flag when a model on a connection starts being answered by a model id that connection had not reported in the previous 14 days — a useful signal to rerun the suite, though it reads metadata only and cannot tell whether answers got worse.