# How many eval cases do you need before switching models?

URL: https://infro.io/blog/how-many-eval-cases-do-you-need-before-switching-models
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-20
Last modified: 2026-11-20
Last reviewed: 2026-11-20 (next review due 2027-05-19)
Topic: Evaluation and models
Format: Methodology

> Size the eval from the margin you can tolerate, not from a round number: roughly 300 cases bound a pass rate near ±4 points, and a paired comparison on identical prompts needs fewer cases than two separate samples. Decide with a confidence interval on the paired difference and a threshold per task type, then confirm the saving in cost per successful task.

Key takeaways:
- Choose a non-inferiority margin first; the number of cases follows from the margin and how often the two models disagree.
- Run both models on the same prompts and analyse the discordant pairs; that needs fewer cases than comparing two independent pass rates.
- Report a Wilson interval on each pass rate and a confidence interval on the paired difference, never a bare percentage.
- Set the pass threshold per task type: tight for deterministic tasks, looser for subjective drafting, and zero-tolerance cases tested separately.
- A lower per-call price can still lose once failed tasks carry a handling cost, so decide on cost per successful task.

## Start from the margin, not the case count

"How many cases?" has no answer until you say how much quality loss you can accept. Sample size is a function of three things: the margin you will tolerate (call it δ, in points of pass rate), how sure you want to be, and how noisy the comparison is. Teams that skip the margin end up with a round number like 100 or 500 that is too small for a tight task and wasteful for a loose one.

The decision you are making is a non-inferiority claim: *the candidate's pass rate is not worse than the current model's by more than δ*. That is different from "the candidate scored 88% and the current one scored 90%". A single score with no interval cannot support either reading. Anthropic's own guidance on evaluation makes a related point: without evals, teams cannot tell real regressions from noise.[1]

Write down four things before you run anything:

1. The task type and its pass criterion (a test, a schema check, a rubric score above a cut-off).
2. The margin δ you can live with, agreed with whoever owns the product outcome.
3. The confidence level, usually 95% two-sided for a confidence interval.
4. The failure cost, which feeds the cost per successful task later.

Everything below turns those four into a case count and a verdict.

## What a single pass rate tells you

For one model on *n* independent cases with *k* passes, use a Wilson interval rather than the plain normal approximation. The plain version misbehaves near 0% and 100%, which is exactly where production pass rates live.

```python
from math import sqrt, comb

def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return centre - half, centre + half

def paired(b, c, n, z=1.96):
    """b: baseline passes, candidate fails. c: candidate passes, baseline fails."""
    diff = (c - b) / n
    se = sqrt(b + c - (b - c) ** 2 / n) / n
    return diff, diff - z * se, diff + z * se

def exact_mcnemar_p(b, c):
    m, k = b + c, min(b, c)
    tail = sum(comb(m, i) for i in range(k + 1)) / 2 ** m
    return min(1.0, 2 * tail)
```

The width of the interval is what you buy with cases. The quick rule is *n* ≈ z²·p(1−p)/E² for a half-width E:

| Expected pass rate | Half-width wanted | Cases (approx.) |
| --- | --- | --- |
| 90% | ±5 points | 139 |
| 90% | ±3 points | 385 |
| 95% | ±3 points | 203 |
| 95% | ±2 points | 456 |

Table: Cases needed to bound one pass rate at 95% confidence. Plain arithmetic from the normal approximation; use the Wilson function for the actual interval.

Two consequences follow. Halving the interval costs four times the cases, so chasing ±1 point on a single model is rarely worth it. And a pass rate near 100% needs fewer cases for the same half-width, but a zero-failure result is weaker than it sounds: zero failures in 300 cases still leaves a 95% upper bound on the failure rate near 1% (the "rule of three", 3/*n*).

## Compare models on the same prompts

Two independent pass rates are the wrong comparison when you can run both models on the identical cases. Items differ in difficulty, and that difficulty is shared: a hard prompt tends to fail on both models. A paired analysis removes that shared variation and looks only at the cases where the models *disagree*.

Build a two-by-two table per case: both pass, both fail, only the baseline passes (b), only the candidate passes (c). Cases where both agree carry no information about which model is better. The difference in pass rate is (c − b)/n, and its standard error depends mostly on b + c, the number of disagreements. An exact McNemar test (a binomial test on b versus c) gives a p-value, and the interval from `paired()` above gives the size of the gap.

The practical payoff is sample size. If the two models disagree on a fraction π of cases, the standard error of the difference is roughly √(π/n). For a one-sided 95% non-inferiority claim with margin δ and 80% power when the true difference is zero:

*n* ≈ (1.645 + 0.84)² · π / δ² ≈ 6.2 · π / δ²

> **Example: Illustrative: paired versus unpaired**
> Take δ = 5 points and a baseline near 90%. If the models disagree on 10% of cases, the formula gives about 247 cases. Comparing two independent samples at the same pass rate needs a variance of about 2·0.9·0.1 = 0.18 per case instead of 0.10, so roughly 445 cases for the same power. The disagreement rate of 10% is assumed; measure yours.

You do not know π in advance. Run both models on a pilot of 50 to 100 cases, count disagreements, and plug that in. If π comes out at 30% (a model change that rewrites many answers), plan for roughly three times the cases of a 10% case.

One caution on repeated runs. Outputs vary between runs, so Anthropic's guidance is to run multiple trials per task.[1] Average the trials into one pass fraction per case before analysis. Ten trials on 30 cases are not 300 independent cases; the case is the unit that counts for the interval.

## Pass thresholds by task type

A single global "95%" is not a threshold. The margin should track how costly a miss is and how objectively you can grade it. Anthropic separates capability evals, which start at a low pass rate and give you something to climb, from regression evals, which should sit near 100% and flag a break when they drop.[1] A model swap is a regression question for work you already ship, so hold the candidate to the regression standard on your existing tasks.

> **Example: Illustrative thresholds and case counts**
> All margins, disagreement rates and case counts below are assumptions to show the arithmetic. Case counts use n ≈ 6.2·π/δ² at 80% power.
> 
> | Task type | Grader | Margin δ | Assumed disagreement π | Cases |
> | --- | --- | --- | --- | --- |
> | Classification, extraction | Exact match | 2 points | 4% | 618 |
> | Tool calls, structured output | Schema and argument checks | 3 points | 8% | 549 |
> | Support replies, summaries | Rubric above cut-off | 5 points | 20% | 494 |
> | Open-ended drafting | Rubric, human spot check | 8 points | 30% | 290 |
> | Multi-step agent task | Final-state check | 5 points | 30% | 741 |
> 
> Table: Illustrative thresholds. Deterministic tasks get tight margins because they are cheap to grade; subjective tasks get looser margins because the grader itself is noisy.

Add two rules on top of the paired test:

- **Absolute floor.** The lower end of the candidate's own Wilson interval must clear the product's minimum pass rate, so a candidate that is "not worse than a model that was already marginal" does not slip through.
- **Zero-tolerance cases.** Prompts where one failure is unacceptable (a refusal that must happen, a field that must never be invented) are tested as a separate list. Report the failure count and the rule-of-three upper bound, and do not average them into the main rate.

If you test several candidates against one baseline, decide the primary comparison before you look at the data. Picking the best of four and testing it as if it were the only one inflates false passes.

## Noise you can mistake for a difference

Binomial interval width is the noise you can compute. There is also noise you cannot. Anthropic found that infrastructure configuration alone moved agentic coding scores by several points, and concluded that leaderboard gaps under 3 percentage points deserve skepticism until the setup is documented and matched. They also note that naive binomial confidence intervals already span 1 to 2 points and that infrastructure effects stack on top of that.[2]

Translate that into your own harness:

- Pin everything except the model: prompt version, sampling settings, tool set, timeouts, retry policy.
- Run baseline and candidate in the same window, not a day apart. Anthropic observed that pass rates anecdotally fluctuate with time of day, likely from API latency, though they did not formally quantify it.[2]
- Count provider errors and timeouts separately from wrong answers. A candidate that fails 3% of calls with a 429 has a capacity problem, not a quality problem, and mixing them hides both.
- Check output truncation. A lower `max_tokens` default on one model turns a correct answer into a failed parse.

If the paired interval straddles your margin after all this, the honest result is "inconclusive". Either add cases (the interval narrows with the square root of *n*) or accept the larger margin knowingly.

## Worked example: cost per successful task

Decisions made on price per call alone go wrong when the lower-priced model fails more often. Use two measures: cost per first-pass success (call cost divided by pass rate) and all-in expected cost per task, which adds the handling cost of a failure.

> **Example: Illustrative numbers, not real prices or measurements**
> A support-reply task, 300 cases, both models run on the same prompts.
> 
> - Current model: $0.0120 per task, 270 of 300 pass (90%).
> - Candidate: $0.0030 per task, 252 of 300 pass (84%).
> - A failed reply costs $0.40 in human rework.
> 
> **Single pass rates.** Candidate Wilson interval: about 79.4% to 87.7%.
> 
> **Paired comparison.** Of the 300 cases, both pass on 240, only the current model passes on 30 (b), only the candidate passes on 12 (c), neither passes on 18. The difference is (12 − 30)/300 = −6.0 points, with a standard error of about 2.1 points and a 95% interval of roughly −10.2 to −1.8 points. The exact test gives a p-value well below 0.05. Against a 5-point margin the candidate is not shown to be non-inferior; the interval's upper edge clears zero in the wrong direction.
> 
> **Cost per first-pass success.** Current: 0.0120 / 0.90 = $0.0133. Candidate: 0.0030 / 0.84 = $0.0036. On this measure the candidate looks far better.
> 
> **All-in expected cost per task.** Current: 0.0120 + 0.10 × 0.40 = $0.0520. Candidate: 0.0030 + 0.16 × 0.40 = $0.0670. Now the candidate is more expensive.
> 
> **Break-even pass rate.** Solve 0.0030 + (1 − p) × 0.40 = 0.0520, so p = 87.75%. The candidate's interval tops out at about 87.7%, so even its optimistic end only reaches break-even.

The lesson is the ordering. Cost per call decides nothing until quality is above the threshold, and when failures carry a real cost, the break-even pass rate is much closer to the baseline than the price ratio suggests. If failures are cheap and automatically retried or escalated, the same arithmetic can favour the candidate; recompute with your own failure cost. A cascade (candidate first, escalate failures to the current model) only works if the failure is detectable, and failures correlate with hard cases, so do not assume the escalated model passes at its overall rate on them.

For the rate side of this formula, use the rates your provider actually bills you. If you have negotiated rates, those replace list rates in the call cost.

## Collecting the cases without a big labeled set

You rarely need hand labels on every case. Three practical ways to get a set you can defend:

- **Deterministic graders first.** Schema validity, exact match on a field, a unit test, a tool call with the right arguments. These cost nothing to run at 600 cases and have no grader noise. Anthropic lists such code-based graders as fast, cheap and reproducible, with the weakness that they are brittle to valid variation.[1]
- **Sampled real prompts.** Draw from production traffic, stratified by task type, with the rare and hard cases deliberately over-represented. A purely random draw underweights the cases most likely to differ between models.
- **A small human-labeled core.** Label 100 to 150 cases by hand for the subjective tasks, and use them to check any model-based grader before it scores the other few hundred. Grader calibration is its own topic; see the related article on judges.

If you run on INFRO, a dataset can be built from logged requests or uploaded as JSONL or CSV, up to 10,000 items, and only requests whose prompts were logged can be added.[3] Be careful with the option that uses the logged answer as the expected output: that makes the *current* model's answer the reference, which measures agreement with the current model, not correctness.

An experiment then runs the dataset across up to six variants, judged by up to five evaluators, with the request count and estimated provider cost per variant shown first and a spend cap it stops at.[4] That estimate gives you the per-task call cost for the cost-per-success formula. Experiments run through the gateway on your own provider accounts, so the spend lands on your provider bill, and your own spend limits apply.[4]

## A decision procedure you can write down

- [ ] Fix the task type, pass criterion and margin δ before looking at results.
- [ ] Run both models on the same cases, in the same window, with everything but the model pinned.
- [ ] Pilot 50 to 100 cases, count disagreements, and size the full run with n ≈ 6.2·π/δ².
- [ ] Average repeated trials into one pass fraction per case.
- [ ] Report a Wilson interval for each model and a paired interval for the difference.
- [ ] Separate provider errors, timeouts and truncation from wrong answers.
- [ ] Test zero-tolerance cases as their own list and report the rule-of-three bound.
- [ ] Compute cost per first-pass success and all-in cost per task with your own failure cost.
- [ ] Switch only if the lower end of the paired interval is above −δ, the absolute floor holds, and the all-in cost is lower.
- [ ] If the result is inconclusive, add cases or keep the current model; do not lower δ after the fact.

That procedure gives a small team a defensible answer from a few hundred cases for most task types. The number you land on is not universal; it comes from your margin, your disagreement rate and your failure cost, and all three should be written down next to the result.

## Questions

### Is 100 eval cases enough to switch models?

Only for a coarse decision. At a 90% pass rate, 100 cases give a 95% Wilson interval of roughly 83% to 94%, a span of about 11 points. That can reject a clearly worse model, but it cannot show a candidate is within 3 or 5 points of the current one.

### Do I need human labels for every case?

Not for tasks with a deterministic check such as schema validity, exact match or unit tests. For subjective tasks, label a smaller set by hand, and calibrate any model grader separately before trusting it on the rest.

### What if the two models disagree on very few cases?

Then the paired difference has a small standard error and you need fewer cases. Count the discordant pairs on a pilot of 50 to 100 prompts, and size the full run from that disagreement rate.

## Sources

1. Demystifying evals for AI agents — Anthropic. https://www.anthropic.com/research/demystifying-evals-for-ai-agents (accessed 2026-11-20)
2. Quantifying infrastructure noise in agentic coding evals — Anthropic. https://www.anthropic.com/research/infrastructure-noise (accessed 2026-11-20)
3. Datasets — INFRO docs. https://infro.io/docs/observability/datasets (accessed 2026-11-20)
4. Experiments — INFRO docs. https://infro.io/docs/observability/experiments (accessed 2026-11-20)
