# Same open-weight model, different hosts: a consistency test

URL: https://infro.io/blog/open-weight-model-hosting-consistency
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Evaluation and models
Format: Methodology

> Treat each host of an open-weight model as a separate system to qualify. Record what the host states about precision and limits, measure run-to-run noise on one host first, then run the same checkable prompts on every host and count only differences larger than that noise.

Open-weight models are attractive partly because many hosts serve them. The same model id on four inference providers looks like four interchangeable suppliers: pick on rate or latency, fail over between them, move when one has a bad week. That is true only if the four behave the same, and nothing guarantees they do.

This article is a test design. It describes what can differ between hosts, a protocol for measuring it, the harness to run it, and how to interpret a difference. It publishes no results, because we did not run it for this article, and any host comparison you read should come with the method and noise band behind it.

## What differs when two hosts serve the same weights

[Diagram: Six places where two hosts serving the same model id can diverge. Only the first is about the weights themselves.]

**Precision.** Hosts routinely serve reduced-precision weights. Together describes quantizing to FP8, FP4 and hybrid precision with calibration and selective mixed precision on sensitive paths, and calls the result near-lossless — its own claim[1]. Fireworks notes that a deployment uses the FP16 checkpoint by default, that each model lists the precisions it has been prepared for, and that quantization can change model numerics and introduce small changes to the output[2]. Groq serves models with its TruePoint numerics, which it says reduce precision only where accuracy is not affected[3]. DeepInfra prints `fp8` or `fp4` on each model listing[4]. Some models also ship quantized by their authors: gpt-oss-120b's MoE weights are MXFP4 in the release itself[5].

**Chat template.** Messages, tool definitions and reasoning settings must be rendered into tokens the way the model was trained. The gpt-oss card says the models should only be used with the harmony response format and will not work correctly otherwise[5]. A host with a subtly different template produces subtly different behaviour, most visibly in tool calls.

**Sampler support and defaults.** Hosts differ in which parameters they honour, their defaults when you send none, and whether `seed` does anything.

**Limits.** A host can serve a shorter context window or lower maximum output than the model card allows. Groq, for example, lists its own context window and maximum output for each model it serves[3].

**Features.** Tool-call parsing and JSON schema enforcement are host software, not weights. One host may support schema-constrained decoding for a model that another serves only in JSON-object mode.

**Runtime nondeterminism.** Even one host is not deterministic. vLLM states it does not guarantee reproducible results by default, for performance, and only provides reproducibility on the same hardware and version[6]. Thinking Machines showed the main cause is kernels that are not batch-invariant, so a request's output depends on server load: 1,000 temperature-0 completions of one prompt produced 80 distinct outputs, all identical for the first 102 tokens[7].

That last point is why most informal host comparisons mislead. If two runs on the same host already differ, a single run on each of two hosts shows noise, not a host difference.

## What hosts publish about precision

[Diagram: What four hosts state publicly about the precision they serve, from their own pages on 2026-09-29. Accuracy claims are the vendors' own.]

Record these statements, with the date, before you test. They tell you where to look first, and they change: a host can requantize a model without changing its name.

## The protocol

[Diagram: The six steps. The baseline step is the one most comparisons skip.]

1. **Pin the model.** One model card, one revision where hosts expose it, and identical request parameters everywhere: temperature 0, the same `max_tokens`, the same system prompt, no host-specific extras.
2. **Record each host's stated configuration**: precision, context window, maximum output, tool and JSON support. If a host does not state precision, record that.
3. **Measure noise first.** Send every prompt five to ten times to one host, spread over time. Compute, per prompt, how often the repeats agree and how the checkable score varies. This is your noise band.
4. **Run the suite on every host**, interleaving hosts in time so a bad hour at one does not look like a quality gap, with the same number of repeats.
5. **Score with checks, not opinions.** Use tasks with a verifiable answer: extraction with a known result, arithmetic, classification with labels, JSON that must validate, tool calls that must name the right function with parseable arguments. Also record `finish_reason`, output tokens and latency.
6. **Report with the method.** Dates, host settings, sample sizes, the noise band and the raw outputs.

Fireworks makes a related point about evaluating quantization: task accuracy scores correctness as a step function, so small shifts in the token distribution are hidden by benchmark noise, and it measures divergence in token probabilities instead[8]. You usually cannot read logits from a hosted API, which is another reason to use many repeats and checkable tasks rather than a single benchmark number.

## The harness

The harness below sends each prompt to several targets, repeats it, checks each answer, and writes one JSON line per response. It uses the OpenAI SDK, so it works against any OpenAI-compatible endpoint; here each target is an INFRO route pinned to one host, described in the next section.

```python
import json
import os
import time
from collections import defaultdict

from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1", api_key=os.environ["INFRO_API_KEY"])

# One route per host, each with a single target: the same model on one connection.
TARGETS = ["qwen-host-a", "qwen-host-b", "qwen-host-c"]
REPEATS = 8
RUN_ID = time.strftime("%Y%m%dT%H%M%S")

# Every case has a checker, so scoring needs no judgement.
CASES = [
    {"id": "sum-1", "prompt": "What is 1847 + 2395? Reply with the number only.",
     "check": lambda text: text.strip() == "4242"},
    {"id": "label-1", "prompt": "Classify the sentiment as positive, negative or neutral. "
     "Reply with one word.\n\nThe refund arrived a week late and nobody answered my emails.",
     "check": lambda text: text.strip().lower() == "negative"},
    {"id": "json-1", "prompt": 'Return JSON only: {"city": string, "year": integer} for '
     '"The treaty was signed in Lisbon in 2007."',
     "check": lambda text: _json_equals(text, {"city": "Lisbon", "year": 2007})},
]


def _json_equals(text: str, expected: dict) -> bool:
    try:
        return json.loads(text) == expected
    except json.JSONDecodeError:
        return False


def run_once(target: str, case: dict, repeat: int) -> dict:
    started = time.monotonic()
    completion = client.chat.completions.create(
        model=target,
        messages=[{"role": "user", "content": case["prompt"]}],
        temperature=0,
        max_tokens=256,
        extra_body={"metadata": {"tags": {"run": RUN_ID, "case": case["id"]}}},
    )
    body = completion.model_dump()
    text = completion.choices[0].message.content or ""
    return {
        "run": RUN_ID, "target": target, "case": case["id"], "repeat": repeat,
        "provider": body.get("provider"), "connection": body.get("connection"),
        "request_id": completion.id, "finish_reason": completion.choices[0].finish_reason,
        "output_tokens": completion.usage.completion_tokens if completion.usage else None,
        "latency_s": round(time.monotonic() - started, 3),
        "passed": case["check"](text), "text": text,
    }


def main() -> None:
    rows = []
    with open(f"consistency-{RUN_ID}.jsonl", "w") as out:
        for repeat in range(REPEATS):
            for case in CASES:
                for target in TARGETS:  # interleave hosts in time
                    row = run_once(target, case, repeat)
                    rows.append(row)
                    out.write(json.dumps(row) + "\n")

    by_target = defaultdict(list)
    agreement = defaultdict(set)
    for row in rows:
        by_target[row["target"]].append(row["passed"])
        agreement[(row["target"], row["case"])].add(row["text"])

    for target, results in by_target.items():
        distinct = [len(agreement[(target, c["id"])]) for c in CASES]
        print(f"{target}: pass {sum(results)}/{len(results)}, "
              f"distinct outputs per case {distinct}")


if __name__ == "__main__":
    main()
```

Three cases are a sketch; a real suite has a hundred or more drawn from production traffic, grouped by task type so a failure can be traced to a capability. Add cases for tool calls with a checker that parses the arguments, and long-context cases that sit near each host's stated limit.

## Reading the results

[Diagram: How to interpret a difference once you have a noise band. Most differences fall into the first three rows.]

> **Example: What a finding looks like**
> Illustrative, not measured: host A passes 796 of 800 checks and host B 771 of 800. Same-host repeats on A disagree on the answer to 1% of cases. B's failures are concentrated in the JSON cases, and every one has `finish_reason` `length` — B serves a lower maximum output than A, and the fix is a larger `max_tokens` on that route, not a different host.

Look for concentration before concluding anything. Failures that cluster in one task type point at a feature or limit difference you can often fix. Failures spread thinly across every task type, beyond the noise band, point at the weights or the template, and are a reason to keep that workload on the hosts that pass. Rerun the suite whenever a host announces an engine change, and on a schedule, because hosts change precision and runtime without changing model names.

## Running the test through INFRO

In INFRO each host is a provider connection on your own account — Together, Fireworks and Groq are connection kinds, and any OpenAI-compatible host can be added as an OpenAI-compatible endpoint. Together and Fireworks model ids carry host-specific prefixes, so each model is named explicitly in the connection's model map. See [providers](https://infro.io/docs/providers).

To pin a request to one host, create one [route](https://infro.io/docs/routes) per host, each with a single target: the model and the connection that should serve it. The harness then calls `qwen-host-a`, `qwen-host-b` and so on. Every response names the `provider` and `connection` that served it, and `route` reads `primary` when no failover happened — check it, because a failover inside a single-target route cannot happen, but a misconfigured route with two targets would silently mix hosts.

The `metadata.tags` in the harness are recorded on each request and never sent to the provider, so the run can be found again in the request log with `GET /v1/requests`, or exported as CSV with `GET /v1/requests/export`, including latency, token counts and estimated provider cost per request. See [request traces](https://infro.io/docs/observability/traces). Estimated provider cost is what your host will bill you, at its list rate or the rate you entered; INFRO's own subscription is separate and does not vary with the host.

Once the test is done, put only the hosts that passed on the production route, in priority order or by weight. INFRO's [traffic changes](https://infro.io/docs/observability/traffic-changes) also flag when a model on a connection starts being answered by a model id that connection had not reported in the previous 14 days — a useful signal to rerun the suite, though it reads metadata only and cannot tell whether answers got worse.

## Questions

### Is an FP8 or FP4 model worse than the BF16 original?

Not necessarily on your task, and not measurably on every task. Hosts describe their low-precision serving as close to the original, but only a test on your own prompts, beyond the run-to-run noise, tells you whether a difference matters for you.

## Sources

1. Together AI delivers fastest inference for the top open-source models — Together AI. https://www.together.ai/blog/fastest-inference-for-the-top-open-source-models (accessed 2026-09-29)
2. Quantization — Fireworks AI. https://docs.fireworks.ai/models/quantization (accessed 2026-09-29)
3. OpenAI GPT-OSS 120B — Groq. https://console.groq.com/docs/model/openai/gpt-oss-120b (accessed 2026-09-29)
4. Models — DeepInfra. https://deepinfra.com/models (accessed 2026-09-29)
5. openai/gpt-oss-120b model card — OpenAI on Hugging Face. https://huggingface.co/openai/gpt-oss-120b (accessed 2026-09-29)
6. Reproducibility — vLLM. https://docs.vllm.ai/en/latest/usage/reproducibility.html (accessed 2026-09-29)
7. Defeating Nondeterminism in LLM Inference — Thinking Machines Lab. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ (accessed 2026-09-29)
8. How Fireworks evaluates quantization precisely and interpretably — Fireworks AI. https://fireworks.ai/blog/fireworks-quantization (accessed 2026-09-29)