Skip to content

Evaluation and modelsMethodology

How many eval cases do you need before switching models?

A lower-priced or newer model only helps if it clears your quality bar, and a small labeled set can't always show that. This method sizes the eval from a margin you choose, compares models on the same prompts, and ends in cost per successful task.

By INFRO Editorial

Published

10 min read

A single amber path on a dark ground forking into two glass branches of slightly different brightness

In short

Size the eval from the margin you can tolerate, not from a round number: roughly 300 cases bound a pass rate near ±4 points, and a paired comparison on identical prompts needs fewer cases than two separate samples. Decide with a confidence interval on the paired difference and a threshold per task type, then confirm the saving in cost per successful task.

Key takeaways

  • Choose a non-inferiority margin first; the number of cases follows from the margin and how often the two models disagree.
  • Run both models on the same prompts and analyse the discordant pairs; that needs fewer cases than comparing two independent pass rates.
  • Report a Wilson interval on each pass rate and a confidence interval on the paired difference, never a bare percentage.
  • Set the pass threshold per task type: tight for deterministic tasks, looser for subjective drafting, and zero-tolerance cases tested separately.
  • A lower per-call price can still lose once failed tasks carry a handling cost, so decide on cost per successful task.

Start from the margin, not the case count

"How many cases?" has no answer until you say how much quality loss you can accept. Sample size is a function of three things: the margin you will tolerate (call it δ, in points of pass rate), how sure you want to be, and how noisy the comparison is. Teams that skip the margin end up with a round number like 100 or 500 that is too small for a tight task and wasteful for a loose one.

The decision you are making is a non-inferiority claim: the candidate's pass rate is not worse than the current model's by more than δ. That is different from "the candidate scored 88% and the current one scored 90%". A single score with no interval cannot support either reading. Anthropic's own guidance on evaluation makes a related point: without evals, teams cannot tell real regressions from noise.1

Write down four things before you run anything:

  1. The task type and its pass criterion (a test, a schema check, a rubric score above a cut-off).
  2. The margin δ you can live with, agreed with whoever owns the product outcome.
  3. The confidence level, usually 95% two-sided for a confidence interval.
  4. The failure cost, which feeds the cost per successful task later.

Everything below turns those four into a case count and a verdict.

What a single pass rate tells you

For one model on n independent cases with k passes, use a Wilson interval rather than the plain normal approximation. The plain version misbehaves near 0% and 100%, which is exactly where production pass rates live.

intervals.py
from math import sqrt, comb

def wilson(k, n, z=1.96):
    p = k / n
    d = 1 + z * z / n
    centre = (p + z * z / (2 * n)) / d
    half = z * sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return centre - half, centre + half

def paired(b, c, n, z=1.96):
    """b: baseline passes, candidate fails. c: candidate passes, baseline fails."""
    diff = (c - b) / n
    se = sqrt(b + c - (b - c) ** 2 / n) / n
    return diff, diff - z * se, diff + z * se

def exact_mcnemar_p(b, c):
    m, k = b + c, min(b, c)
    tail = sum(comb(m, i) for i in range(k + 1)) / 2 ** m
    return min(1.0, 2 * tail)

The width of the interval is what you buy with cases. The quick rule is n ≈ z²·p(1−p)/E² for a half-width E:

Expected pass rateHalf-width wantedCases (approx.)
90%±5 points139
90%±3 points385
95%±3 points203
95%±2 points456
Cases needed to bound one pass rate at 95% confidence. Plain arithmetic from the normal approximation; use the Wilson function for the actual interval.

Two consequences follow. Halving the interval costs four times the cases, so chasing ±1 point on a single model is rarely worth it. And a pass rate near 100% needs fewer cases for the same half-width, but a zero-failure result is weaker than it sounds: zero failures in 300 cases still leaves a 95% upper bound on the failure rate near 1% (the "rule of three", 3/n).

Compare models on the same prompts

Two independent pass rates are the wrong comparison when you can run both models on the identical cases. Items differ in difficulty, and that difficulty is shared: a hard prompt tends to fail on both models. A paired analysis removes that shared variation and looks only at the cases where the models disagree.

Build a two-by-two table per case: both pass, both fail, only the baseline passes (b), only the candidate passes (c). Cases where both agree carry no information about which model is better. The difference in pass rate is (c − b)/n, and its standard error depends mostly on b + c, the number of disagreements. An exact McNemar test (a binomial test on b versus c) gives a p-value, and the interval from paired() above gives the size of the gap.

The practical payoff is sample size. If the two models disagree on a fraction π of cases, the standard error of the difference is roughly √(π/n). For a one-sided 95% non-inferiority claim with margin δ and 80% power when the true difference is zero:

n ≈ (1.645 + 0.84)² · π / δ² ≈ 6.2 · π / δ²

You do not know π in advance. Run both models on a pilot of 50 to 100 cases, count disagreements, and plug that in. If π comes out at 30% (a model change that rewrites many answers), plan for roughly three times the cases of a 10% case.

One caution on repeated runs. Outputs vary between runs, so Anthropic's guidance is to run multiple trials per task.1 Average the trials into one pass fraction per case before analysis. Ten trials on 30 cases are not 300 independent cases; the case is the unit that counts for the interval.

Pass thresholds by task type

A single global "95%" is not a threshold. The margin should track how costly a miss is and how objectively you can grade it. Anthropic separates capability evals, which start at a low pass rate and give you something to climb, from regression evals, which should sit near 100% and flag a break when they drop.1 A model swap is a regression question for work you already ship, so hold the candidate to the regression standard on your existing tasks.

Add two rules on top of the paired test:

  • Absolute floor. The lower end of the candidate's own Wilson interval must clear the product's minimum pass rate, so a candidate that is "not worse than a model that was already marginal" does not slip through.
  • Zero-tolerance cases. Prompts where one failure is unacceptable (a refusal that must happen, a field that must never be invented) are tested as a separate list. Report the failure count and the rule-of-three upper bound, and do not average them into the main rate.

If you test several candidates against one baseline, decide the primary comparison before you look at the data. Picking the best of four and testing it as if it were the only one inflates false passes.

Noise you can mistake for a difference

Binomial interval width is the noise you can compute. There is also noise you cannot. Anthropic found that infrastructure configuration alone moved agentic coding scores by several points, and concluded that leaderboard gaps under 3 percentage points deserve skepticism until the setup is documented and matched. They also note that naive binomial confidence intervals already span 1 to 2 points and that infrastructure effects stack on top of that.2

Translate that into your own harness:

  • Pin everything except the model: prompt version, sampling settings, tool set, timeouts, retry policy.
  • Run baseline and candidate in the same window, not a day apart. Anthropic observed that pass rates anecdotally fluctuate with time of day, likely from API latency, though they did not formally quantify it.2
  • Count provider errors and timeouts separately from wrong answers. A candidate that fails 3% of calls with a 429 has a capacity problem, not a quality problem, and mixing them hides both.
  • Check output truncation. A lower max_tokens default on one model turns a correct answer into a failed parse.

If the paired interval straddles your margin after all this, the honest result is "inconclusive". Either add cases (the interval narrows with the square root of n) or accept the larger margin knowingly.

Worked example: cost per successful task

Decisions made on price per call alone go wrong when the lower-priced model fails more often. Use two measures: cost per first-pass success (call cost divided by pass rate) and all-in expected cost per task, which adds the handling cost of a failure.

The lesson is the ordering. Cost per call decides nothing until quality is above the threshold, and when failures carry a real cost, the break-even pass rate is much closer to the baseline than the price ratio suggests. If failures are cheap and automatically retried or escalated, the same arithmetic can favour the candidate; recompute with your own failure cost. A cascade (candidate first, escalate failures to the current model) only works if the failure is detectable, and failures correlate with hard cases, so do not assume the escalated model passes at its overall rate on them.

For the rate side of this formula, use the rates your provider actually bills you. If you have negotiated rates, those replace list rates in the call cost.

Collecting the cases without a big labeled set

You rarely need hand labels on every case. Three practical ways to get a set you can defend:

  • Deterministic graders first. Schema validity, exact match on a field, a unit test, a tool call with the right arguments. These cost nothing to run at 600 cases and have no grader noise. Anthropic lists such code-based graders as fast, cheap and reproducible, with the weakness that they are brittle to valid variation.1
  • Sampled real prompts. Draw from production traffic, stratified by task type, with the rare and hard cases deliberately over-represented. A purely random draw underweights the cases most likely to differ between models.
  • A small human-labeled core. Label 100 to 150 cases by hand for the subjective tasks, and use them to check any model-based grader before it scores the other few hundred. Grader calibration is its own topic; see the related article on judges.

If you run on INFRO, a dataset can be built from logged requests or uploaded as JSONL or CSV, up to 10,000 items, and only requests whose prompts were logged can be added.3 Be careful with the option that uses the logged answer as the expected output: that makes the current model's answer the reference, which measures agreement with the current model, not correctness.

An experiment then runs the dataset across up to six variants, judged by up to five evaluators, with the request count and estimated provider cost per variant shown first and a spend cap it stops at.4 That estimate gives you the per-task call cost for the cost-per-success formula. Experiments run through the gateway on your own provider accounts, so the spend lands on your provider bill, and your own spend limits apply.4

A decision procedure you can write down

  • Fix the task type, pass criterion and margin δ before looking at results.
  • Run both models on the same cases, in the same window, with everything but the model pinned.
  • Pilot 50 to 100 cases, count disagreements, and size the full run with n ≈ 6.2·π/δ².
  • Average repeated trials into one pass fraction per case.
  • Report a Wilson interval for each model and a paired interval for the difference.
  • Separate provider errors, timeouts and truncation from wrong answers.
  • Test zero-tolerance cases as their own list and report the rule-of-three bound.
  • Compute cost per first-pass success and all-in cost per task with your own failure cost.
  • Switch only if the lower end of the paired interval is above −δ, the absolute floor holds, and the all-in cost is lower.
  • If the result is inconclusive, add cases or keep the current model; do not lower δ after the fact.

That procedure gives a small team a defensible answer from a few hundred cases for most task types. The number you land on is not universal; it comes from your margin, your disagreement rate and your failure cost, and all three should be written down next to the result.

Questions readers ask

Is 100 eval cases enough to switch models?
Only for a coarse decision. At a 90% pass rate, 100 cases give a 95% Wilson interval of roughly 83% to 94%, a span of about 11 points. That can reject a clearly worse model, but it cannot show a candidate is within 3 or 5 points of the current one.
Do I need human labels for every case?
Not for tasks with a deterministic check such as schema validity, exact match or unit tests. For subjective tasks, label a smaller set by hand, and calibrate any model grader separately before trusting it on the rest.
What if the two models disagree on very few cases?
Then the paired difference has a small standard error and you need fewer cases. Count the discordant pairs on a pilot of 50 to 100 prompts, and size the full run from that disagreement rate.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Datasets

    Available with a plan

    Inputs and expected outputs built from logged requests in one click, uploaded as JSONL or CSV, or sent from code, kept until you delete them. On every plan.

  • Experiments

    Available with a plan

    A dataset run through the gateway across up to six variants (a model or route, a prompt version, settings), judged by your evaluators and compared side by side, with the request count and estimated provider cost shown before it starts and a spend cap it stops at. On every plan.

  • Evaluators

    Available with a plan

    A rubric a chat model judges answers by, on your own provider account: a sample of live traffic held to a monthly spend limit, or up to 1,000 logged requests on demand with an estimate first and a spend cap. Each verdict is a score on the request it judged, and each judge call is a request on your provider, counted against your allowance. On every plan.

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Demystifying evals for AI agents (opens in a new tab) — Anthropic. Accessed Nov 20, 2026.
  2. 2.Quantifying infrastructure noise in agentic coding evals (opens in a new tab) — Anthropic. Accessed Nov 20, 2026.
  3. 3.Datasets (opens in a new tab) — INFRO docs. Accessed Nov 20, 2026.
  4. 4.Experiments (opens in a new tab) — INFRO docs. Accessed Nov 20, 2026.

About this article

Read Anthropic's engineering pages on agent evals and on infrastructure noise in evals, and INFRO's Datasets and Experiments docs, on 2026-11-20. The OpenAI index page returned HTTP 403 and was not used. Interval formulas and sample-size arithmetic are standard statistics worked by hand here; every pass rate, price and case count in the examples is illustrative, not measured.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.