# How to evaluate LLMs on real production workloads

URL: https://infro.io/blog/evaluate-llms-real-workloads
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-08-23
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Evaluation and models
Format: Guide

> Evaluate models on a sample of your own production tasks with a pass criterion per task, run each candidate several times on identical inputs, and record success rate, cost per successful task, latency and malformed-output rate together. Public leaderboards are useful for shortlisting; the decision comes from your traffic, and it has to be re-run when models, prompts or providers change.

Key takeaways:
- Sample real tasks, keep the difficult tail, and version the dataset together with the prompt.
- Give every task a programmatic pass criterion where possible; audit any model-based judge, which has documented biases.
- Run each candidate several times on identical inputs and record success, cost per successful task, latency and malformed output together.
- Translate results into a routing matrix: default model, eligible substitutes, escalation, and workloads where fallback is forbidden.
- Re-run on every prompt, model or provider change, and on a schedule, because hosted models change without notice.
- After rollout, keep scoring a small sample of production answers in the background so a regression shows up between offline runs.

Model comparisons age faster than almost anything else written about AI. A table of prices and context windows is out of date within weeks, a leaderboard ranking describes someone else's tasks, and impressions of which model "feels better at tool calling" rarely survive contact with your own schemas. What lasts is the method: a way to find out, on your traffic, which model does the job at an acceptable cost — and to find out again when something changes.

## What public benchmarks can and cannot tell you

Public benchmarks are useful for building a shortlist. They show whether a model is in the right class for a kind of task, and they catch obvious gaps. They cannot tell you how a model handles your prompts, your tools, your documents, your latency budget or your failure costs, because they were built from someone else's distribution of tasks. When several candidates score close together on a public suite, the differences between them on your workload are the only differences that matter.

Use benchmarks to decide what to test. Use your own evaluation to decide what to ship.

## Build the dataset from real traffic

1. **Sample from production**, following your privacy process for any customer data. A few hundred carefully chosen tasks per workload is usually more informative than thousands of synthetic prompts.
2. **Stratify.** Include ordinary requests, difficult ones, malformed inputs, very long inputs and safety-sensitive cases in deliberate proportions. Models tend to separate in the tail, not at the median.
3. **Record the expected outcome** for each task — a correct label, a valid structure, a test that must pass — at the time you sample it.
4. **Version the dataset together with the prompt.** Otherwise a prompt improvement is indistinguishable from a model improvement.
5. **Grow it from incidents.** Every production failure that could have been caught becomes a new case.

## Define success per task

The grader should match the task. Where a program can decide, let it.

| Task type | Grader | Watch for |
| --- | --- | --- |
| Extraction and classification | Exact match or schema validation against the expected value | Labels that are ambiguous in the source data |
| Code generation | Unit tests executed in isolation | Tests that pass trivially |
| Retrieval-grounded answers | Citation present and supported by the retrieved passage | Answers correct by luck without grounding |
| Tool use and agents | The tool sequence completes and the final state is correct | Success on a mocked tool that behaves unlike the real one |
| Open-ended writing | Rubric scored by people, or by a model judge audited against people | Judge bias |

Table: Match the grader to the task, and prefer a program to a judge when a program can decide.

Model-based judges are convenient and imperfect. Research on using models as judges documents position bias, verbosity bias and self-enhancement bias[1]. Mitigate them: randomize the order of candidates, score against a rubric rather than asking which answer is better, use a judge from a different model family than the candidates, and hand-check a sample of verdicts before trusting the rest.

A newer option for rubric grading is a decision model: instead of writing a verdict in prose, it answers a typed question — yes or no, one of several labels, or a level on a rubric — with a probability. That makes the threshold explicit and the grader's uncertainty visible, though the same audit against human labels still applies. The engines that exist and how they differ are in [decision models compared](https://infro.io/blog/decision-models-compared-2026).

## Run it properly

- **Identical inputs.** Every candidate receives the same prompts, tools and parameters. Tune prompts per model only after comparing defaults, and only for the top candidates.
- **Repeated runs.** Run each task several times. A single run measures one sample of a random process.
- **Fixed parameters.** Record temperature and any reasoning or output-length settings; changing them changes the result.
- **Record operational failures, not just scores.** A model that produces the right answer in invalid JSON fails in production.

For each candidate and workload, keep five numbers:

| Metric | Why it is on the list |
| --- | --- |
| Success rate | The share of tasks that met the pass criterion |
| Cost per successful task | Total cost of all attempts divided by successful tasks |
| p95 total latency and time to first token | Whether the model fits the experience's time budget |
| Malformed-output rate | Invalid structure or tool arguments; predicts on-call work |
| Variance across runs | How much the success rate moves between identical runs |

Table: The minimum a model decision should be based on.

Two operational details decide whether the numbers mean anything. First, **turn off every response cache** between the harness and the model. An exact-match cache replays the first answer to an identical request, so three runs at temperature 0 become one run counted three times and the variance column reads zero. Provider prompt caching is different: it reuses the processing of a prompt prefix, not the answer, and is safe to leave on.

Second, scheduled evaluation runs have no user waiting, so they fit batch pricing where a provider offers it: OpenAI prices its Batch API at half the cost of its synchronous APIs, with a 24-hour completion window[2], and Anthropic charges batch usage at 50% of standard prices[3].

## Cost per successful task

The comparison that decides most model choices combines the first two metrics.

> **Example: Why the pass rate belongs in the cost (hypothetical rates)**
> A candidate costs $0.002 per attempt and passes 80% of tasks. Retrying failures, each success takes 1.25 attempts on average, so a success costs $0.0025. The incumbent costs $0.010 per attempt and passes 97%, about $0.0103 per success. The candidate wins — unless its failures reach users unnoticed, in which case the cost of a failure is not the price of a retry but the cost of the mistake. Put a price on that before deciding.

How this figure fits into wider cost control, in order of risk, is in the [cost optimization playbook](https://infro.io/blog/llm-cost-optimization-playbook).

## A minimal harness

The harness below runs every candidate on every task several times through one OpenAI-compatible endpoint and records the numbers above. It assumes each task carries its own `check` function; replace the evaluation logic with your graders.

```python
import os
import statistics
import time
from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1", api_key=os.environ["INFRO_API_KEY"])

CANDIDATES = ["anthropic/claude-sonnet-5.5", "openai/gpt-6-sol", "openai/gpt-6-luna"]
RUNS = 3

def run(model, task):
    started = time.monotonic()
    response = client.chat.completions.create(
        model=model,
        messages=task["messages"],
        temperature=0,
        extra_body={
            "metadata": {"tags": {"eval_run": task["run_id"]}},
            "cache": {"mode": "bypass"},  # every run must reach the model
        },
    )
    elapsed = time.monotonic() - started
    text = response.choices[0].message.content or ""
    return {
        "passed": task["check"](text),
        "cost": response.usage.model_extra.get("cost"),  # estimated provider cost; None if unpriced
        "seconds": elapsed,
    }

def evaluate(tasks):
    for model in CANDIDATES:
        results = [run(model, task) for task in tasks for _ in range(RUNS)]
        passed = [r for r in results if r["passed"]]
        costs = [r["cost"] for r in results if r["cost"] is not None]
        print(model, {
            "success_rate": len(passed) / len(results),
            "cost_per_success": (sum(costs) / len(passed)) if passed and costs else None,
            "p95_seconds": statistics.quantiles([r["seconds"] for r in results], n=20)[18],
        })
```

Only the model string changes between candidates, so the harness stays the same when the list does. The `cache` field is INFRO's per-request control for its response cache; against another endpoint, use whatever switch its cache offers, or drop the line if there is no cache.

## Turn results into a routing matrix

An evaluation is finished when it has changed a decision someone can read. Write the result down per workload:

| Workload | Default model | Eligible substitutes | Escalate to | Fallback to a different model |
| --- | --- | --- | --- | --- |
| Ticket classification | The passing model with the lowest cost per successful task | Any other model that passed | None | Allowed among passing models |
| Reply drafting | The model with the best pass rate within budget | Same model on another account | A stronger model on validation failure | Only models that passed the drafting evaluation |
| Refund decisions | The highest-accuracy model | Same model on another account only | A person | Forbidden |

Table: The artifact an evaluation produces. Routing and failover configuration should be derived from it.

When the evaluation shows that one workload splits by difficulty — short lookups pass on a small model, multi-step requests only on a large one — the matrix needs a row per class and something that decides the class per request. That can be a rule on request properties, or a small classifier that labels the prompt before routing, with a timeout and a default class so routing never waits on it. The trade-offs of putting a classifier in the request path are in [classifiers in the request path](https://infro.io/blog/classifiers-in-the-request-path). Evaluate the classifier too: a misrouted hard prompt fails on the small model, and that failure belongs in the cost per successful task.

## When to re-run

- Every prompt, retrieval or tool change for the workload.
- Every model version change, including a provider's silent update to a hosted model.
- Every change of provider or account serving the model.
- On a fixed schedule, because hosted behaviour drifts.

Roll a winning change out gradually: send a small share of production traffic to it and compare against the unchanged path before widening it[4].

Between offline runs, two lighter checks keep watch. **Score a sample in production**: after an answer has been returned, grade a small share of real prompt-and-answer pairs with the same rubric, in the background, so the user never waits for the grader and a drop in the mean score per model shows up within days rather than at the next scheduled run. **Watch for silent changes**: record the model id the provider says answered each request, and alert when a new id appears or when output length, finish reasons or the cached share of input move sharply. Neither replaces the evaluation set; both tell you when to run it.

## Evaluations in INFRO: tagged runs, routes and sampled scores

The harness above runs unchanged against every model your connected providers serve, because INFRO keeps one OpenAI-compatible request shape across them, and every call runs on your own provider account. Each response includes `usage.cost`, an estimate of what your provider will bill — `null` when the model is unpriced, never 0 — and each request is stored with its model, provider, latency and token counts. Tagging requests with an evaluation run id in `metadata` lets you pull exactly that run's records from `GET /v1/requests` with a `tag.eval_run` filter and compute per-model figures from them.

INFRO's [response cache](https://infro.io/docs/features/caching) is off unless an owner or admin turns it on; where it is on, `"cache": {"mode": "bypass"}` sends the request to the provider without reading or writing an entry, which is what the harness sets.

Once the matrix is written, [routes](https://infro.io/docs/routes) express it as configuration rather than code: targets in priority order or by weight, if-rules on request properties, and a prompt classifier whose class (`class`, `class_confidence`) a rule can read, with a `fallback_class` used on a timeout, an error or low confidence.

The two production checks have counterparts too. An **eval monitor** is a decision policy with a `monitor` block and a `sample_rate` from 0.001 to 1; after an answer has been returned, it scores a sampled share of chat completions in the background on a decision model you connect, records each score as its own row under the request it judged, and shows the mean score per served model on the policy's page ([Decisions API](https://infro.io/docs/api/decisions)). [Traffic changes](https://infro.io/docs/observability/traffic-changes) raises a finding when a model is answered by a new snapshot, or when output tokens, the cached share of prompt tokens, fallbacks or length-limited answers move past documented thresholds. It reads request metadata only, so it can say that something changed, not that answers got worse; that is what the evaluation set is for.

## Sources

1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023) — arXiv. https://arxiv.org/abs/2306.05685 (accessed 2026-10-13)
2. Batch API — OpenAI. https://developers.openai.com/api/docs/guides/batch (accessed 2026-10-13)
3. Batch processing — Anthropic. https://platform.claude.com/docs/en/build-with-claude/batch-processing (accessed 2026-10-13)
4. Canarying Releases — The Site Reliability Workbook — Google. https://sre.google/workbook/canarying-releases/ (accessed 2026-10-13)

## Revision history

- 2026-10-13: Reviewed against every source. The harness now bypasses response caching, which would otherwise replay one answer across repeated runs, and uses current model ids. Added model-based graders that return probabilities, classifier routing as a way to apply a routing matrix per request, and sampled background scoring and snapshot-change detection as the checks that run between offline evaluations. Reworded the batch-pricing note.
- 2026-09-11: Expanded into a full guide and merged the method from the retired comparison "GPT-5.1 vs Claude vs Gemini vs DeepSeek: price-performance", which now redirects here. Its model price table, context-window figures and first-person performance impressions were removed because they could not be sourced or kept current. Added grader guidance, a runnable harness, a routing matrix and cited sources.