Model comparisons age faster than almost anything else written about AI. A table of prices and context windows is out of date within weeks, a leaderboard ranking describes someone else's tasks, and impressions of which model "feels better at tool calling" rarely survive contact with your own schemas. What lasts is the method: a way to find out, on your traffic, which model does the job at an acceptable cost — and to find out again when something changes.
What public benchmarks can and cannot tell you
Public benchmarks are useful for building a shortlist. They show whether a model is in the right class for a kind of task, and they catch obvious gaps. They cannot tell you how a model handles your prompts, your tools, your documents, your latency budget or your failure costs, because they were built from someone else's distribution of tasks. When several candidates score close together on a public suite, the differences between them on your workload are the only differences that matter.
Use benchmarks to decide what to test. Use your own evaluation to decide what to ship.
Build the dataset from real traffic
- Sample from production, following your privacy process for any customer data. A few hundred carefully chosen tasks per workload is usually more informative than thousands of synthetic prompts.
- Stratify. Include ordinary requests, difficult ones, malformed inputs, very long inputs and safety-sensitive cases in deliberate proportions. Models tend to separate in the tail, not at the median.
- Record the expected outcome for each task — a correct label, a valid structure, a test that must pass — at the time you sample it.
- Version the dataset together with the prompt. Otherwise a prompt improvement is indistinguishable from a model improvement.
- Grow it from incidents. Every production failure that could have been caught becomes a new case.
Define success per task
The grader should match the task. Where a program can decide, let it.
| Task type | Grader | Watch for |
|---|---|---|
| Extraction and classification | Exact match or schema validation against the expected value | Labels that are ambiguous in the source data |
| Code generation | Unit tests executed in isolation | Tests that pass trivially |
| Retrieval-grounded answers | Citation present and supported by the retrieved passage | Answers correct by luck without grounding |
| Tool use and agents | The tool sequence completes and the final state is correct | Success on a mocked tool that behaves unlike the real one |
| Open-ended writing | Rubric scored by people, or by a model judge audited against people | Judge bias |
Model-based judges are convenient and imperfect. Research on using models as judges documents position bias, verbosity bias and self-enhancement bias1. Mitigate them: randomize the order of candidates, score against a rubric rather than asking which answer is better, use a judge from a different model family than the candidates, and hand-check a sample of verdicts before trusting the rest.
A newer option for rubric grading is a decision model: instead of writing a verdict in prose, it answers a typed question — yes or no, one of several labels, or a level on a rubric — with a probability. That makes the threshold explicit and the grader's uncertainty visible, though the same audit against human labels still applies. The engines that exist and how they differ are in decision models compared.
Run it properly
- Identical inputs. Every candidate receives the same prompts, tools and parameters. Tune prompts per model only after comparing defaults, and only for the top candidates.
- Repeated runs. Run each task several times. A single run measures one sample of a random process.
- Fixed parameters. Record temperature and any reasoning or output-length settings; changing them changes the result.
- Record operational failures, not just scores. A model that produces the right answer in invalid JSON fails in production.
For each candidate and workload, keep five numbers:
| Metric | Why it is on the list |
|---|---|
| Success rate | The share of tasks that met the pass criterion |
| Cost per successful task | Total cost of all attempts divided by successful tasks |
| p95 total latency and time to first token | Whether the model fits the experience's time budget |
| Malformed-output rate | Invalid structure or tool arguments; predicts on-call work |
| Variance across runs | How much the success rate moves between identical runs |
Two operational details decide whether the numbers mean anything. First, turn off every response cache between the harness and the model. An exact-match cache replays the first answer to an identical request, so three runs at temperature 0 become one run counted three times and the variance column reads zero. Provider prompt caching is different: it reuses the processing of a prompt prefix, not the answer, and is safe to leave on.
Second, scheduled evaluation runs have no user waiting, so they fit batch pricing where a provider offers it: OpenAI prices its Batch API at half the cost of its synchronous APIs, with a 24-hour completion window2, and Anthropic charges batch usage at 50% of standard prices3.
Cost per successful task
The comparison that decides most model choices combines the first two metrics.
How this figure fits into wider cost control, in order of risk, is in the cost optimization playbook.
A minimal harness
The harness below runs every candidate on every task several times through one OpenAI-compatible endpoint and records the numbers above. It assumes each task carries its own check function; replace the evaluation logic with your graders.
import os
import statistics
import time
from openai import OpenAI
client = OpenAI(base_url="https://api.infro.io/v1", api_key=os.environ["INFRO_API_KEY"])
CANDIDATES = ["anthropic/claude-sonnet-5.5", "openai/gpt-6-sol", "openai/gpt-6-luna"]
RUNS = 3
def run(model, task):
started = time.monotonic()
response = client.chat.completions.create(
model=model,
messages=task["messages"],
temperature=0,
extra_body={
"metadata": {"tags": {"eval_run": task["run_id"]}},
"cache": {"mode": "bypass"}, # every run must reach the model
},
)
elapsed = time.monotonic() - started
text = response.choices[0].message.content or ""
return {
"passed": task["check"](text),
"cost": response.usage.model_extra.get("cost"), # estimated provider cost; None if unpriced
"seconds": elapsed,
}
def evaluate(tasks):
for model in CANDIDATES:
results = [run(model, task) for task in tasks for _ in range(RUNS)]
passed = [r for r in results if r["passed"]]
costs = [r["cost"] for r in results if r["cost"] is not None]
print(model, {
"success_rate": len(passed) / len(results),
"cost_per_success": (sum(costs) / len(passed)) if passed and costs else None,
"p95_seconds": statistics.quantiles([r["seconds"] for r in results], n=20)[18],
})Only the model string changes between candidates, so the harness stays the same when the list does. The cache field is INFRO's per-request control for its response cache; against another endpoint, use whatever switch its cache offers, or drop the line if there is no cache.
Turn results into a routing matrix
An evaluation is finished when it has changed a decision someone can read. Write the result down per workload:
| Workload | Default model | Eligible substitutes | Escalate to | Fallback to a different model |
|---|---|---|---|---|
| Ticket classification | The passing model with the lowest cost per successful task | Any other model that passed | None | Allowed among passing models |
| Reply drafting | The model with the best pass rate within budget | Same model on another account | A stronger model on validation failure | Only models that passed the drafting evaluation |
| Refund decisions | The highest-accuracy model | Same model on another account only | A person | Forbidden |
When the evaluation shows that one workload splits by difficulty — short lookups pass on a small model, multi-step requests only on a large one — the matrix needs a row per class and something that decides the class per request. That can be a rule on request properties, or a small classifier that labels the prompt before routing, with a timeout and a default class so routing never waits on it. The trade-offs of putting a classifier in the request path are in classifiers in the request path. Evaluate the classifier too: a misrouted hard prompt fails on the small model, and that failure belongs in the cost per successful task.
When to re-run
- Every prompt, retrieval or tool change for the workload.
- Every model version change, including a provider's silent update to a hosted model.
- Every change of provider or account serving the model.
- On a fixed schedule, because hosted behaviour drifts.
Roll a winning change out gradually: send a small share of production traffic to it and compare against the unchanged path before widening it4.
Between offline runs, two lighter checks keep watch. Score a sample in production: after an answer has been returned, grade a small share of real prompt-and-answer pairs with the same rubric, in the background, so the user never waits for the grader and a drop in the mean score per model shows up within days rather than at the next scheduled run. Watch for silent changes: record the model id the provider says answered each request, and alert when a new id appears or when output length, finish reasons or the cached share of input move sharply. Neither replaces the evaluation set; both tell you when to run it.
Evaluations in INFRO: tagged runs, routes and sampled scores
The harness above runs unchanged against every model your connected providers serve, because INFRO keeps one OpenAI-compatible request shape across them, and every call runs on your own provider account. Each response includes usage.cost, an estimate of what your provider will bill — null when the model is unpriced, never 0 — and each request is stored with its model, provider, latency and token counts. Tagging requests with an evaluation run id in metadata lets you pull exactly that run's records from GET /v1/requests with a tag.eval_run filter and compute per-model figures from them.
INFRO's response cache is off unless an owner or admin turns it on; where it is on, "cache": {"mode": "bypass"} sends the request to the provider without reading or writing an entry, which is what the harness sets.
Once the matrix is written, routes express it as configuration rather than code: targets in priority order or by weight, if-rules on request properties, and a prompt classifier whose class (class, class_confidence) a rule can read, with a fallback_class used on a timeout, an error or low confidence.
The two production checks have counterparts too. An eval monitor is a decision policy with a monitor block and a sample_rate from 0.001 to 1; after an answer has been returned, it scores a sampled share of chat completions in the background on a decision model you connect, records each score as its own row under the request it judged, and shows the mean score per served model on the policy's page (Decisions API). Traffic changes raises a finding when a model is answered by a new snapshot, or when output tokens, the cached share of prompt tokens, fallbacks or length-limited answers move past documented thresholds. It reads request metadata only, so it can say that something changed, not that answers got worse; that is what the evaluation set is for.