A/B testing LLM models in production: splits, metrics, sample size

How do you A/B test LLM models in production? Pass an offline evaluation first. Then split users — not individual requests — between the current model and the candidate with a stable hash, send each arm to its model, and compare a short list of metrics fixed in advance: task success, user signals, p95 latency, error rate and cost per successful task. Ramp the candidate up in steps with automatic stop conditions, and run until each arm has enough users to detect the change you care about.

Leaderboards tell you how a model does on someone else's tasks. A production test tells you how it does on yours, with your prompts, your users and your tools.

Before any traffic: the offline gate

OpenAI's guidance is to adopt eval-driven development, evaluating early and often and growing the eval set from production over time¹. Anthropic's advice for building those tests: prefer more questions with automated grading over fewer hand-graded ones, structure questions so they can be graded automatically, and use a different model to grade than the one that generated the output².

The gate is simple: the candidate must match or beat the incumbent on your offline set, including your known hard cases, before it sees a user. An online test is for confirming a result, not for discovering that the new model ignores your JSON schema. Evaluating LLMs on real workloads covers building the set.

One adjustment before the comparison: prompts are usually tuned for the incumbent. Give the candidate an honest prompt pass first, or you are testing your prompt's fit rather than the model.

Assignment: users, not requests

Split by user (or account, or conversation) in almost every product. A person who sees one model on one message and another on the next gives you signals you cannot attribute, and multi-turn context makes the arms leak into each other. Requests are a fine unit only for independent, stateless calls such as classification.

Make the assignment deterministic, so it survives restarts and needs no storage:

import hashlib

def arm(user_id: str, experiment: str, candidate_share: float) -> str:
    """Stable assignment: the same user always lands in the same arm."""
    digest = hashlib.sha256(f"{experiment}:{user_id}".encode()).digest()
    bucket = int.from_bytes(digest[:8], "big") / 2**64   # uniform in [0, 1)
    return "candidate" if bucket < candidate_share else "control"

Including the experiment name in the hash means each experiment gets an independent split, so the same users are not always the ones who receive every new model.

The metrics that decide

Fix these before the test, with the direction and size of change that would make you ship:

OpenAI recommends calibrating automated scoring against human feedback so it stays trustworthy¹; a model-graded quality score that nobody has checked against people is a number, not a measure.

Cost per successful task is the one teams forget. A candidate with a lower rate per token that needs more retries, longer outputs or more escalations can cost more per finished job. Cost per successful task shows the arithmetic.

How many users you need

For a success-rate metric, a standard approximation for 80% power at 5% significance is:

n ≈ 16 × p × (1 − p) / δ²
# p = baseline success rate, δ = the absolute difference you want to detect

Two consequences. Small products cannot detect small differences online, so lean harder on offline evaluation and graded samples. And stopping the test the first time the dashboard looks good inflates false positives; decide the sample size first and look at the end.

Ramp, with automatic stops

1. 1% for a day. Catches integration breakage: errors, timeouts, schema failures.
2. 5%. First read on latency and cost under real load.
3. 25%, then 50%. Enough traffic to read the decision metrics.

At every step, guardrail metrics stop the test automatically: error rate above a threshold, p95 latency above a ceiling, a cost per request far above the estimate. A test that needs a human to notice a regression will run all weekend.

Traps specific to LLM tests

• Cold caches. A new model starts with no prompt cache, so its first days cost more and can be slower than steady state. Compare cost from the point the cache hit rate has settled.
• Tokenizers. Different models count the same prompt differently, so compare cost in dollars, never tokens.
• Novelty. Users react to a different writing style for a few days. Run at least one full week and check whether the effect holds in the second week.
• Shared limits. Both arms may share provider rate limits; a busy arm can throttle the other. Watch 429s per arm.
• Shadow mode is not free. Running the candidate on every request in the background without showing it doubles model cost for the test's duration. Use it on a sample.

Reading the result

When the test reaches its sample size, read it in a fixed order, so the result does not depend on which chart someone opened first:

1. Guardrails. Did error rate, refusals or p95 latency cross a limit at any point? If so, the candidate fails regardless of quality.
2. Primary metric. Is the difference in task success outside the noise? Report the confidence interval, not just the point estimate.
3. Cost per successful task. Compute it per arm from real spend and real success. A small quality gain at a large cost increase is a business decision, not a statistical one.
4. Segments. Check the result by plan, language and request size. A model that wins on average can lose badly for one customer segment, and that segment may be your largest account.
5. Durability. Compare week one with week two. An effect that fades is usually novelty.

Write the decision down with the numbers that made it, then make the winning arm the default and remove the experiment's code path. A finished experiment left in the code is a routing rule nobody remembers.

Splitting traffic with INFRO

INFRO routes provide two ways to split traffic. A weighted route picks the first target at random in proportion to its weights and still fails over to the others, which suits a canary of independent requests. For a per-user experiment, keep the assignment in your app and send the arm in the request's metadata (for example "arm": "candidate"); a route rule that matches metadata.arm sends those requests to the candidate's targets and everything else to the default. Each request's record names the route and the connection that served it, with latency, status and an estimated provider cost, so cost and errors per arm come from the request log. Eval monitors can score a share of served chat answers against a rubric on a decision model you connect.

Summary

Gate on an offline evaluation, split by user with a stable hash, decide on metrics chosen in advance — cost per successful task among them — size the test before you start, ramp with automatic stops, and wait out cache warm-up and novelty before you call it.