Skip to content

Cost and FinOpsGuide

What should an eval run cost? Budgeting your evaluation spend

Eval and replay runs show up on the provider bill with no budget behind them. This guide sizes the cost of one run, separates it from production spend, and sets caps at three layers. The worked numbers are illustrative, so replace them with your own token counts.

By INFRO Editorial

Published

12 min read

Glowing token streams being compressed into a precise efficient AI infrastructure pipeline

In short

An eval run costs cases × variants × repeats × the answer call, plus one judge call per evaluator on every answer, plus retries. Estimate it before it starts, tag it as its own workflow in its own environment, and cap it per run, per evaluator and per project.

Key takeaways

  • One eval run is the answer calls (cases × variants × repeats) plus a judge call per evaluator on every answer, so judge spend scales with the number of evaluators and can match the answer spend.
  • Reasoning tokens, retries and repeated samples are multipliers that a case count alone hides; put each in the estimate as an explicit factor.
  • Tag every eval request with its own workflow and run it in its own environment, because an experiment started without an environment lands in production by default.
  • Cap at three layers: a per-run cap at or above the estimate, a monthly limit on live judging, and a blocking project or key budget that bounds everything else.
  • Budget evals as a line with an owner, a forecast built from planned runs, and a holdback, not as unallocated overhead in the production line.

Why eval spend ends up unowned

Production traffic has a feature, a team and a revenue line behind it. An eval run has none of those. It is started by an engineer comparing two prompts, by a CI job replaying last week's traffic, or by a sampler that judges a share of live answers, and it arrives on the provider bill as ordinary model spend. Nobody forecast it, so nobody is accountable for it.

The FinOps Framework treats budgeting as approved funding for planned technology activity, tracked against actuals, with owners for each scope of work.1 Evaluation is planned activity. You can list the runs you intend to do in a quarter, so you can forecast and budget for them like any other workload. What is missing is usually three things: a way to size one run, a way to see eval spend apart from production, and a ceiling that stops a run that went wrong.

This guide covers those three in order. It assumes you already know how many cases you need and which models you want to compare; for that, see how many eval cases you need before switching models and cost per successful task. Here the question is only what a run costs and how to keep it inside a number.

The cost formula for one run

A run has two kinds of calls. The first is the answer call: each dataset case sent to each variant under test. The second is the judge call: each answer scored by each evaluator. INFRO's experiments estimate counts them the same way: items × variants for the answers, plus one judge call per evaluator on every answer.2

Text
answers      = cases × variants × repeats
judge_calls  = answers × evaluators
run_cost     = answers × answer_cost × retry_factor
             + judge_calls × judge_cost

Each term hides a decision:

  • cases is your dataset size. It is the term you chose on statistical grounds, so it is the one you should change last.
  • variants is every model, prompt version or settings combination you compare. Include the current production configuration as a variant, or you have nothing to compare against.
  • repeats is how many times you sample each case per variant. One is cheapest; more than one measures run-to-run variance and multiplies both answer and judge spend.
  • evaluators is how many rubrics score each answer. Each added rubric is another full judge call per answer.
  • answer_cost is input tokens × input rate + output tokens × output rate, where output includes any reasoning tokens your model bills as output (see below).
  • retry_factor is a multiplier slightly above 1.0 for calls you re-send after a timeout or 429.
  • judge_cost is the judge prompt (rubric, the case input, the answer, and the expected output) plus the judge's verdict.

Two details of INFRO's evaluators matter when you size the judge term. Each substituted value in a rubric is clipped to 12,000 characters, and the judge is asked for JSON at temperature 0 with at most 400 output tokens.3 So a judge call has a known output ceiling, but its input is the largest part of it: the rubric plus the full question and the full answer.

What the multipliers do to the number

Four things inflate a run beyond what the case count suggests.

Reasoning tokens. A thinking model spends tokens before it answers. If your provider bills those at the output rate, a short visible answer can be a long billed one, and output is the expensive side of the rate card. Check how your model reports and bills reasoning, and size the answer call from the billed output, not the visible text. Reasoning tokens: the hidden cost of thinking models covers the mechanics.

Judge model choice. The judge reads more than the answerer did, because it gets the question, the answer and often a reference. Putting a frontier model in that seat can make judging cost as much as answering. The judge should be the smallest model that agrees with human labels on your rubric, which you establish by calibration rather than assumption; see can an LLM judge be trusted.

Retries. A retried call re-sends the whole prompt. A run against a rate-limited key can retry a large share of its calls, and on a long prompt that is real spend. Keep the retry factor as a visible number in your estimate, and revisit it after your first run using the actual request count against the planned one.

Repeats and variants. These multiply everything. Going from three variants to six doubles the run, and going from one sample to three triples it. Both are cheap to add in a config file and expensive on the bill.

A worked example

The point of the table is the shape, not the dollars. In the first row, the judge costs nearly as much as the thing being judged. In the third row, turning reasoning off cuts the answer cost by more than half, which only helps if reasoning does not change the answers you are trying to measure. Decide that with a small pilot, not by assumption.

Get the estimate from the system, not a spreadsheet

A spreadsheet drifts from the real request shape. Prefer an estimate computed from the same dataset, variants and evaluators the run will use. INFRO's experiments accept dry_run: true and return the request count, the estimated provider cost per variant, per evaluator and in total, and the share of your monthly request allowance the run would use. A model with no known rate is listed as unpriced and the total is null rather than $0.2

estimate-first.sh
curl -X POST "https://api.infro.io/manage/v1/experiments" \
  -H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset": "support-golden",
    "variants": [
      { "label": "current",   "model": "anthropic/claude-sonnet-5" },
      { "label": "candidate", "model": "anthropic/claude-sonnet-5",
        "prompt": { "slug": "support-reply", "version": 4 } }
    ],
    "evaluators": ["helpfulness"],
    "environment": "evaluation",
    "dry_run": true
  }'

Read the estimate against two checks before approving the run. First, confirm the judge-call count equals answers × evaluators; if it does not, an evaluator is filtered in a way you did not intend. Second, confirm nothing is unpriced. Estimates use the vendor's reference rate unless you have entered the rates your contracts carry, and INFRO lets you enter those per model so the estimate uses them first.5 If you have a negotiated rate, enter it before you size runs, or every estimate misses in the same direction.

The estimate is still an estimate. The request log records the model, usage and estimated provider cost of each request, and it is operational evidence, not an invoice for model usage.6 Reconcile against your provider's own cost report on a schedule.

Your dataset itself is a cost lever. Datasets can be built from logged requests or uploaded as JSONL or CSV; a smaller set that concentrates on the cases that discriminate between variants costs less per run than a large random sample, and longer cases cost more per case, so trimming the input term helps too.

Separate eval spend from production

A provider invoice cannot tell an eval request from a production one. Both are the same model on the same key. The separation has to be made before the call, in three places.

The FinOps Framework defines a scope as a segment of spending aligned to business constructs such as products, cost centers or environments,7 and evaluation fits that definition: it is a segment you can name and report on, if you tag it.

  1. A separate environment. Run evals in an environment of their own rather than production. INFRO environments let you add your own beside production, staging and development, and each gateway key belongs to exactly one. An experiment has an environment setting whose default is production,2 so an experiment that omits it lands in your production figures. Set it explicitly in every script and CI job.
  2. A workflow name and tags per run. Attribution adds the dimensions only your application knows: workflow, feature, trace_id, step and free-form tags.8 Name the workflow for the purpose, such as eval-support-golden, and put the experiment or release in a tag.
  3. A promoted tag for grouping. Tags stay on the request until you promote their key; an organization can promote up to five, and only requests metered after promotion carry the key as a dimension.8 Promote the one you want to group eval spend by before the first run you care about, because nothing is backfilled.
tag_eval_calls.py
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1",
                api_key=os.environ["INFRO_EVAL_KEY"])  # a key in the evaluation environment

resp = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",
    messages=[{"role": "user", "content": case["input"]}],
    extra_body={"metadata": {
        "workflow": "eval-support-golden",
        "trace_id": run_id,
        "step": "answer",
        "tags": {"release": "2026-11-candidate"},
    }},
)

Once calls carry a workflow and a trace id, each run is listed as a unit with its requests, errors, duration, models and estimated cost. The runs endpoint filters by workflow and sorts by cost, over a window of up to 31 days.9

Shell
curl "https://api.infro.io/v1/runs?workflow=eval-support-golden&sort=costliest" \
  -H "Authorization: Bearer $INFRO_API_KEY"

That gives you the number a FinOps owner needs: what each eval run actually cost, set against its estimate. Group by feature or a promoted tag in the usage endpoint for the monthly rollup.8

Cap the run at three layers

An estimate is a prediction. A cap is what holds when the prediction is wrong because a dataset doubled, a model was swapped for a pricier one, or a loop re-submitted the run. Use more than one layer, because each catches a different failure.

LayerWhat it boundsWhere it stops
Per-run capOne experimentThe run stops at the cap
Evaluator monthly limitLive judging of production samplesJudging pauses until next month
Project or key budgetEverything in the eval environmentBlocking budget refuses the request
Three cap layers and what each one bounds.

Per-run. To start an experiment you send the same body without dry_run and with a spend_cap in US dollars that is at least the estimate; a lower cap is refused. An experiment with an unpriced model cannot start, because its spend could not be held to the cap.2 Set the cap to the estimate plus a small margin you can justify, not a round number three times higher.

Live judging. If you judge a sample of live traffic, the evaluator needs a monthly_spend_limit in estimated provider dollars per calendar month (UTC). When its judge calls reach it, live judging pauses until the next month, and calls already in flight still finish.3 That pause is a quiet failure, so alert on it: a paused evaluator means your quality signal has stopped.

Project and key budgets. The outer layer is a blocking budget on the eval project or on the eval key. INFRO supports organization, project, member and key budgets; a key limit always blocks, while a project budget can alert or block.10 A blocking budget admits a request only if its worst-case estimated cost fits the room left, so close to the limit a request can be refused briefly for room that is reserved but not yet spent.10 Leave headroom between the per-run cap and the project budget, or your cap will not be what ends the run.

Judge calls and experiment calls both run as the key that made them, under that key's model access, guardrails, environment and spend limits, so a limit on the eval key governs both.3 That is the reason to give evals their own key rather than reuse an application key.

Put the budget in the FinOps cycle

A cap stops spend; a budget explains it. The FinOps Framework's "shift left" guidance for budgeting says to set response thresholds with enough headroom for billing-data delays and alert evaluation intervals, to give each threshold an owner and a response time, and to test the response before relying on it.1 Apply that to eval spend:

  1. List the runs you plan for the quarter: release gates, model-switch evaluations, drift checks, and any live judging. Multiply each by its estimate.
  2. Add a holdback for unplanned runs. The framework describes holdback as money not allocated to a specific purpose, held by the budget owner for changes and forecasting errors.1 Evals are exactly where unplanned runs happen, such as a new model launch or a retirement.
  3. Assign the line to the team that decides what ships, and set alerts at thresholds you choose on the project budget.
  4. Review the forecast against actuals monthly. A run that cost twice its estimate points to a wrong token assumption, a retry storm or an unpriced model, and each of those is fixable.

For the finance reader, a short monthly view is enough: planned runs, completed runs, estimated spend against budget, and the largest run. The usage and runs endpoints and the FOCUS export carry the data, so eval spend can sit beside the rest of your cloud cost rather than in a separate spreadsheet.

A checklist before the next run

  • The run has a purpose and an owner, and appears in the quarter's list of planned runs.
  • The estimate was computed from the real dataset, variants and evaluators, and nothing is unpriced.
  • Negotiated rates are entered, so the estimate is not on list rates by accident.
  • The judge model was chosen by calibration, and its output length is bounded.
  • Reasoning on or off is a recorded choice, with a pilot showing it does not change the answers.
  • Variants and repeats are the minimum that answers the question.
  • The environment is set explicitly and is not production.
  • Every call carries a workflow name and a trace id, and the grouping tag is promoted.
  • The per-run cap is at least the estimate, and the eval key and project have blocking budgets above it.
  • Live judging has a monthly limit, and an alert fires if the limit is reached.
  • After the run, actual cost from the runs list is compared with the estimate and the gap is explained.

Doing this on your own provider accounts

If your provider accounts are connected to INFRO, the pieces above are available together: experiments with an estimate and a spend cap, evaluators with a monthly limit, environments, attribution, runs and budgets. The model spend stays on your own provider accounts at your own rates; INFRO takes 0% of it, and the subscription pays for the control plane.5 You can build the same controls yourself with a wrapper around your provider SDK, but the estimate, the cap and the tagging then become code you maintain.

Questions readers ask

Does the estimate for a run match what my provider invoices?
Not exactly. A gateway estimate is built from token counts and a rate table, and the request log is operational evidence rather than an invoice. Enter your negotiated rates if you have them, and reconcile the estimate against the provider's own cost report on a schedule so a persistent gap gets investigated.
Who should own the eval budget?
The team that decides which model or prompt ships should own it, with finance setting the envelope. The FinOps budgeting capability describes budgets allocated to owners of a specific scope of work, with a holdback for forecasting errors.1 An eval line with a named owner is easier to defend than a share of the production line.
Should evals run on a batch tier to lower the rate?
Possibly, for runs nobody is waiting on. Anthropic's pricing page lists batch processing at half the standard rate.4 The trade-off is latency and a separate execution path, so a run started through a gateway experiment feature will not use it unless that path supports it; check before you plan around the lower rate.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Experiments

    Available with a plan

    A dataset run through the gateway across up to six variants (a model or route, a prompt version, settings), judged by your evaluators and compared side by side, with the request count and estimated provider cost shown before it starts and a spend cap it stops at. On every plan.

  • Evaluators

    Available with a plan

    A rubric a chat model judges answers by, on your own provider account: a sample of live traffic held to a monthly spend limit, or up to 1,000 logged requests on demand with an estimate first and a spend cap. Each verdict is a score on the request it judged, and each judge call is a request on your provider, counted against your allowance. On every plan.

  • Datasets

    Available with a plan

    Inputs and expected outputs built from logged requests in one click, uploaded as JSONL or CSV, or sent from code, kept until you delete them. On every plan.

  • Budgets and spend ceilings

    Available with a plan

    A hard cap per organization, project, member, and key, with alerts at your own thresholds.

    Read the documentation for Budgets and spend ceilings
  • Spend attribution

    Available with a plan

    Group cost by model, project, key, or an end-user id you attach to the request.

    Read the documentation for Spend attribution
  • Environments

    Available with a plan

    Production, staging and development to start, and your own beside them. A connection or route serves one, several or every environment; each gateway key belongs to exactly one, and never reaches a connection outside it.

    Read the documentation for Environments
  • Runs

    Available with a plan

    Requests that share a trace id listed as one run each: requests, errors, duration, models, estimated provider cost, scores and prompt versions, filtered and sorted over a window of up to 31 days. In the API, the console, the management API and MCP, on every plan.

  • Usage and cost per request

    Available with a plan

    Every request records its model, latency, token or unit counts, and estimated provider cost, queryable by project and key.

    Read the documentation for Usage and cost per request

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Budgeting FinOps Framework Capability (opens in a new tab) — FinOps Foundation. Accessed Nov 21, 2026.
  2. 2.Experiments — Observability (opens in a new tab) — INFRO. Accessed Nov 21, 2026.
  3. 3.Evaluators — Observability (opens in a new tab) — INFRO. Accessed Nov 21, 2026.
  4. 4.Plans & pricing (API rates) (opens in a new tab) — Anthropic. Accessed Nov 21, 2026.
  5. 5.INFRO documentation overview (opens in a new tab) — INFRO. Accessed Nov 21, 2026.
  6. 6.Request traces — Observability (opens in a new tab) — INFRO. Accessed Nov 21, 2026.
  7. 7.FinOps Framework Overview (opens in a new tab) — FinOps Foundation. Accessed Nov 21, 2026.
  8. 8.Cost attribution — Cost & finance (opens in a new tab) — INFRO. Accessed Nov 21, 2026.
  9. 9.Runs — Observability (opens in a new tab) — INFRO. Accessed Nov 21, 2026.
  10. 10.Spend controls — Organizations (opens in a new tab) — INFRO. Accessed Nov 21, 2026.

About this article

Read on 2026-11-21: the FinOps Framework overview and its Budgeting capability page, Anthropic's API pricing page, and INFRO's docs overview plus its Experiments, Evaluators, Spend controls, Runs, Request traces and Cost attribution pages. OpenAI's pricing page returned only business-plan seat pricing with no API rates, so no OpenAI rate is quoted; worked-example numbers are illustrative and use Anthropic's list rates.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.