Start from the margin, not the case count
"How many cases?" has no answer until you say how much quality loss you can accept. Sample size is a function of three things: the margin you will tolerate (call it δ, in points of pass rate), how sure you want to be, and how noisy the comparison is. Teams that skip the margin end up with a round number like 100 or 500 that is too small for a tight task and wasteful for a loose one.
The decision you are making is a non-inferiority claim: the candidate's pass rate is not worse than the current model's by more than δ. That is different from "the candidate scored 88% and the current one scored 90%". A single score with no interval cannot support either reading. Anthropic's own guidance on evaluation makes a related point: without evals, teams cannot tell real regressions from noise.1
Write down four things before you run anything:
- The task type and its pass criterion (a test, a schema check, a rubric score above a cut-off).
- The margin δ you can live with, agreed with whoever owns the product outcome.
- The confidence level, usually 95% two-sided for a confidence interval.
- The failure cost, which feeds the cost per successful task later.
Everything below turns those four into a case count and a verdict.
What a single pass rate tells you
For one model on n independent cases with k passes, use a Wilson interval rather than the plain normal approximation. The plain version misbehaves near 0% and 100%, which is exactly where production pass rates live.
from math import sqrt, comb
def wilson(k, n, z=1.96):
p = k / n
d = 1 + z * z / n
centre = (p + z * z / (2 * n)) / d
half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return centre - half, centre + half
def paired(b, c, n, z=1.96):
"""b: baseline passes, candidate fails. c: candidate passes, baseline fails."""
diff = (c - b) / n
se = sqrt(b + c - (b - c) ** 2 / n) / n
return diff, diff - z * se, diff + z * se
def exact_mcnemar_p(b, c):
m, k = b + c, min(b, c)
tail = sum(comb(m, i) for i in range(k + 1)) / 2 ** m
return min(1.0, 2 * tail)The width of the interval is what you buy with cases. The quick rule is n ≈ z²·p(1−p)/E² for a half-width E:
| Expected pass rate | Half-width wanted | Cases (approx.) |
|---|---|---|
| 90% | ±5 points | 139 |
| 90% | ±3 points | 385 |
| 95% | ±3 points | 203 |
| 95% | ±2 points | 456 |
Two consequences follow. Halving the interval costs four times the cases, so chasing ±1 point on a single model is rarely worth it. And a pass rate near 100% needs fewer cases for the same half-width, but a zero-failure result is weaker than it sounds: zero failures in 300 cases still leaves a 95% upper bound on the failure rate near 1% (the "rule of three", 3/n).
Compare models on the same prompts
Two independent pass rates are the wrong comparison when you can run both models on the identical cases. Items differ in difficulty, and that difficulty is shared: a hard prompt tends to fail on both models. A paired analysis removes that shared variation and looks only at the cases where the models disagree.
Build a two-by-two table per case: both pass, both fail, only the baseline passes (b), only the candidate passes (c). Cases where both agree carry no information about which model is better. The difference in pass rate is (c − b)/n, and its standard error depends mostly on b + c, the number of disagreements. An exact McNemar test (a binomial test on b versus c) gives a p-value, and the interval from paired() above gives the size of the gap.
The practical payoff is sample size. If the two models disagree on a fraction π of cases, the standard error of the difference is roughly √(π/n). For a one-sided 95% non-inferiority claim with margin δ and 80% power when the true difference is zero:
n ≈ (1.645 + 0.84)² · π / δ² ≈ 6.2 · π / δ²
You do not know π in advance. Run both models on a pilot of 50 to 100 cases, count disagreements, and plug that in. If π comes out at 30% (a model change that rewrites many answers), plan for roughly three times the cases of a 10% case.
One caution on repeated runs. Outputs vary between runs, so Anthropic's guidance is to run multiple trials per task.1 Average the trials into one pass fraction per case before analysis. Ten trials on 30 cases are not 300 independent cases; the case is the unit that counts for the interval.
Pass thresholds by task type
A single global "95%" is not a threshold. The margin should track how costly a miss is and how objectively you can grade it. Anthropic separates capability evals, which start at a low pass rate and give you something to climb, from regression evals, which should sit near 100% and flag a break when they drop.1 A model swap is a regression question for work you already ship, so hold the candidate to the regression standard on your existing tasks.
Add two rules on top of the paired test:
- Absolute floor. The lower end of the candidate's own Wilson interval must clear the product's minimum pass rate, so a candidate that is "not worse than a model that was already marginal" does not slip through.
- Zero-tolerance cases. Prompts where one failure is unacceptable (a refusal that must happen, a field that must never be invented) are tested as a separate list. Report the failure count and the rule-of-three upper bound, and do not average them into the main rate.
If you test several candidates against one baseline, decide the primary comparison before you look at the data. Picking the best of four and testing it as if it were the only one inflates false passes.
Noise you can mistake for a difference
Binomial interval width is the noise you can compute. There is also noise you cannot. Anthropic found that infrastructure configuration alone moved agentic coding scores by several points, and concluded that leaderboard gaps under 3 percentage points deserve skepticism until the setup is documented and matched. They also note that naive binomial confidence intervals already span 1 to 2 points and that infrastructure effects stack on top of that.2
Translate that into your own harness:
- Pin everything except the model: prompt version, sampling settings, tool set, timeouts, retry policy.
- Run baseline and candidate in the same window, not a day apart. Anthropic observed that pass rates anecdotally fluctuate with time of day, likely from API latency, though they did not formally quantify it.2
- Count provider errors and timeouts separately from wrong answers. A candidate that fails 3% of calls with a 429 has a capacity problem, not a quality problem, and mixing them hides both.
- Check output truncation. A lower
max_tokensdefault on one model turns a correct answer into a failed parse.
If the paired interval straddles your margin after all this, the honest result is "inconclusive". Either add cases (the interval narrows with the square root of n) or accept the larger margin knowingly.
Worked example: cost per successful task
Decisions made on price per call alone go wrong when the lower-priced model fails more often. Use two measures: cost per first-pass success (call cost divided by pass rate) and all-in expected cost per task, which adds the handling cost of a failure.
The lesson is the ordering. Cost per call decides nothing until quality is above the threshold, and when failures carry a real cost, the break-even pass rate is much closer to the baseline than the price ratio suggests. If failures are cheap and automatically retried or escalated, the same arithmetic can favour the candidate; recompute with your own failure cost. A cascade (candidate first, escalate failures to the current model) only works if the failure is detectable, and failures correlate with hard cases, so do not assume the escalated model passes at its overall rate on them.
For the rate side of this formula, use the rates your provider actually bills you. If you have negotiated rates, those replace list rates in the call cost.
Collecting the cases without a big labeled set
You rarely need hand labels on every case. Three practical ways to get a set you can defend:
- Deterministic graders first. Schema validity, exact match on a field, a unit test, a tool call with the right arguments. These cost nothing to run at 600 cases and have no grader noise. Anthropic lists such code-based graders as fast, cheap and reproducible, with the weakness that they are brittle to valid variation.1
- Sampled real prompts. Draw from production traffic, stratified by task type, with the rare and hard cases deliberately over-represented. A purely random draw underweights the cases most likely to differ between models.
- A small human-labeled core. Label 100 to 150 cases by hand for the subjective tasks, and use them to check any model-based grader before it scores the other few hundred. Grader calibration is its own topic; see the related article on judges.
If you run on INFRO, a dataset can be built from logged requests or uploaded as JSONL or CSV, up to 10,000 items, and only requests whose prompts were logged can be added.3 Be careful with the option that uses the logged answer as the expected output: that makes the current model's answer the reference, which measures agreement with the current model, not correctness.
An experiment then runs the dataset across up to six variants, judged by up to five evaluators, with the request count and estimated provider cost per variant shown first and a spend cap it stops at.4 That estimate gives you the per-task call cost for the cost-per-success formula. Experiments run through the gateway on your own provider accounts, so the spend lands on your provider bill, and your own spend limits apply.4
A decision procedure you can write down
- Fix the task type, pass criterion and margin δ before looking at results.
- Run both models on the same cases, in the same window, with everything but the model pinned.
- Pilot 50 to 100 cases, count disagreements, and size the full run with n ≈ 6.2·π/δ².
- Average repeated trials into one pass fraction per case.
- Report a Wilson interval for each model and a paired interval for the difference.
- Separate provider errors, timeouts and truncation from wrong answers.
- Test zero-tolerance cases as their own list and report the rule-of-three bound.
- Compute cost per first-pass success and all-in cost per task with your own failure cost.
- Switch only if the lower end of the paired interval is above −δ, the absolute floor holds, and the all-in cost is lower.
- If the result is inconclusive, add cases or keep the current model; do not lower δ after the fact.
That procedure gives a small team a defensible answer from a few hundred cases for most task types. The number you land on is not universal; it comes from your margin, your disagreement rate and your failure cost, and all three should be written down next to the result.
