Why eval spend ends up unowned
Production traffic has a feature, a team and a revenue line behind it. An eval run has none of those. It is started by an engineer comparing two prompts, by a CI job replaying last week's traffic, or by a sampler that judges a share of live answers, and it arrives on the provider bill as ordinary model spend. Nobody forecast it, so nobody is accountable for it.
The FinOps Framework treats budgeting as approved funding for planned technology activity, tracked against actuals, with owners for each scope of work.1 Evaluation is planned activity. You can list the runs you intend to do in a quarter, so you can forecast and budget for them like any other workload. What is missing is usually three things: a way to size one run, a way to see eval spend apart from production, and a ceiling that stops a run that went wrong.
This guide covers those three in order. It assumes you already know how many cases you need and which models you want to compare; for that, see how many eval cases you need before switching models and cost per successful task. Here the question is only what a run costs and how to keep it inside a number.
The cost formula for one run
A run has two kinds of calls. The first is the answer call: each dataset case sent to each variant under test. The second is the judge call: each answer scored by each evaluator. INFRO's experiments estimate counts them the same way: items × variants for the answers, plus one judge call per evaluator on every answer.2
answers = cases × variants × repeats
judge_calls = answers × evaluators
run_cost = answers × answer_cost × retry_factor
+ judge_calls × judge_costEach term hides a decision:
- cases is your dataset size. It is the term you chose on statistical grounds, so it is the one you should change last.
- variants is every model, prompt version or settings combination you compare. Include the current production configuration as a variant, or you have nothing to compare against.
- repeats is how many times you sample each case per variant. One is cheapest; more than one measures run-to-run variance and multiplies both answer and judge spend.
- evaluators is how many rubrics score each answer. Each added rubric is another full judge call per answer.
- answer_cost is input tokens × input rate + output tokens × output rate, where output includes any reasoning tokens your model bills as output (see below).
- retry_factor is a multiplier slightly above 1.0 for calls you re-send after a timeout or 429.
- judge_cost is the judge prompt (rubric, the case input, the answer, and the expected output) plus the judge's verdict.
Two details of INFRO's evaluators matter when you size the judge term. Each substituted value in a rubric is clipped to 12,000 characters, and the judge is asked for JSON at temperature 0 with at most 400 output tokens.3 So a judge call has a known output ceiling, but its input is the largest part of it: the rubric plus the full question and the full answer.
What the multipliers do to the number
Four things inflate a run beyond what the case count suggests.
Reasoning tokens. A thinking model spends tokens before it answers. If your provider bills those at the output rate, a short visible answer can be a long billed one, and output is the expensive side of the rate card. Check how your model reports and bills reasoning, and size the answer call from the billed output, not the visible text. Reasoning tokens: the hidden cost of thinking models covers the mechanics.
Judge model choice. The judge reads more than the answerer did, because it gets the question, the answer and often a reference. Putting a frontier model in that seat can make judging cost as much as answering. The judge should be the smallest model that agrees with human labels on your rubric, which you establish by calibration rather than assumption; see can an LLM judge be trusted.
Retries. A retried call re-sends the whole prompt. A run against a rate-limited key can retry a large share of its calls, and on a long prompt that is real spend. Keep the retry factor as a visible number in your estimate, and revisit it after your first run using the actual request count against the planned one.
Repeats and variants. These multiply everything. Going from three variants to six doubles the run, and going from one sample to three triples it. Both are cheap to add in a config file and expensive on the bill.
A worked example
The point of the table is the shape, not the dollars. In the first row, the judge costs nearly as much as the thing being judged. In the third row, turning reasoning off cuts the answer cost by more than half, which only helps if reasoning does not change the answers you are trying to measure. Decide that with a small pilot, not by assumption.
Get the estimate from the system, not a spreadsheet
A spreadsheet drifts from the real request shape. Prefer an estimate computed from the same dataset, variants and evaluators the run will use. INFRO's experiments accept dry_run: true and return the request count, the estimated provider cost per variant, per evaluator and in total, and the share of your monthly request allowance the run would use. A model with no known rate is listed as unpriced and the total is null rather than $0.2
curl -X POST "https://api.infro.io/manage/v1/experiments" \
-H "Authorization: Bearer $INFRO_MANAGEMENT_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"dataset": "support-golden",
"variants": [
{ "label": "current", "model": "anthropic/claude-sonnet-5" },
{ "label": "candidate", "model": "anthropic/claude-sonnet-5",
"prompt": { "slug": "support-reply", "version": 4 } }
],
"evaluators": ["helpfulness"],
"environment": "evaluation",
"dry_run": true
}'Read the estimate against two checks before approving the run. First, confirm the judge-call count equals answers × evaluators; if it does not, an evaluator is filtered in a way you did not intend. Second, confirm nothing is unpriced. Estimates use the vendor's reference rate unless you have entered the rates your contracts carry, and INFRO lets you enter those per model so the estimate uses them first.5 If you have a negotiated rate, enter it before you size runs, or every estimate misses in the same direction.
The estimate is still an estimate. The request log records the model, usage and estimated provider cost of each request, and it is operational evidence, not an invoice for model usage.6 Reconcile against your provider's own cost report on a schedule.
Your dataset itself is a cost lever. Datasets can be built from logged requests or uploaded as JSONL or CSV; a smaller set that concentrates on the cases that discriminate between variants costs less per run than a large random sample, and longer cases cost more per case, so trimming the input term helps too.
Separate eval spend from production
A provider invoice cannot tell an eval request from a production one. Both are the same model on the same key. The separation has to be made before the call, in three places.
The FinOps Framework defines a scope as a segment of spending aligned to business constructs such as products, cost centers or environments,7 and evaluation fits that definition: it is a segment you can name and report on, if you tag it.
- A separate environment. Run evals in an environment of their own rather than production. INFRO environments let you add your own beside production, staging and development, and each gateway key belongs to exactly one. An experiment has an
environmentsetting whose default is production,2 so an experiment that omits it lands in your production figures. Set it explicitly in every script and CI job. - A workflow name and tags per run. Attribution adds the dimensions only your application knows:
workflow,feature,trace_id,stepand free-form tags.8 Name the workflow for the purpose, such aseval-support-golden, and put the experiment or release in a tag. - A promoted tag for grouping. Tags stay on the request until you promote their key; an organization can promote up to five, and only requests metered after promotion carry the key as a dimension.8 Promote the one you want to group eval spend by before the first run you care about, because nothing is backfilled.
import os
from openai import OpenAI
client = OpenAI(base_url="https://api.infro.io/v1",
api_key=os.environ["INFRO_EVAL_KEY"]) # a key in the evaluation environment
resp = client.chat.completions.create(
model="anthropic/claude-sonnet-5",
messages=[{"role": "user", "content": case["input"]}],
extra_body={"metadata": {
"workflow": "eval-support-golden",
"trace_id": run_id,
"step": "answer",
"tags": {"release": "2026-11-candidate"},
}},
)Once calls carry a workflow and a trace id, each run is listed as a unit with its requests, errors, duration, models and estimated cost. The runs endpoint filters by workflow and sorts by cost, over a window of up to 31 days.9
curl "https://api.infro.io/v1/runs?workflow=eval-support-golden&sort=costliest" \
-H "Authorization: Bearer $INFRO_API_KEY"That gives you the number a FinOps owner needs: what each eval run actually cost, set against its estimate. Group by feature or a promoted tag in the usage endpoint for the monthly rollup.8
Cap the run at three layers
An estimate is a prediction. A cap is what holds when the prediction is wrong because a dataset doubled, a model was swapped for a pricier one, or a loop re-submitted the run. Use more than one layer, because each catches a different failure.
| Layer | What it bounds | Where it stops |
|---|---|---|
| Per-run cap | One experiment | The run stops at the cap |
| Evaluator monthly limit | Live judging of production samples | Judging pauses until next month |
| Project or key budget | Everything in the eval environment | Blocking budget refuses the request |
Per-run. To start an experiment you send the same body without dry_run and with a spend_cap in US dollars that is at least the estimate; a lower cap is refused. An experiment with an unpriced model cannot start, because its spend could not be held to the cap.2 Set the cap to the estimate plus a small margin you can justify, not a round number three times higher.
Live judging. If you judge a sample of live traffic, the evaluator needs a monthly_spend_limit in estimated provider dollars per calendar month (UTC). When its judge calls reach it, live judging pauses until the next month, and calls already in flight still finish.3 That pause is a quiet failure, so alert on it: a paused evaluator means your quality signal has stopped.
Project and key budgets. The outer layer is a blocking budget on the eval project or on the eval key. INFRO supports organization, project, member and key budgets; a key limit always blocks, while a project budget can alert or block.10 A blocking budget admits a request only if its worst-case estimated cost fits the room left, so close to the limit a request can be refused briefly for room that is reserved but not yet spent.10 Leave headroom between the per-run cap and the project budget, or your cap will not be what ends the run.
Judge calls and experiment calls both run as the key that made them, under that key's model access, guardrails, environment and spend limits, so a limit on the eval key governs both.3 That is the reason to give evals their own key rather than reuse an application key.
Put the budget in the FinOps cycle
A cap stops spend; a budget explains it. The FinOps Framework's "shift left" guidance for budgeting says to set response thresholds with enough headroom for billing-data delays and alert evaluation intervals, to give each threshold an owner and a response time, and to test the response before relying on it.1 Apply that to eval spend:
- List the runs you plan for the quarter: release gates, model-switch evaluations, drift checks, and any live judging. Multiply each by its estimate.
- Add a holdback for unplanned runs. The framework describes holdback as money not allocated to a specific purpose, held by the budget owner for changes and forecasting errors.1 Evals are exactly where unplanned runs happen, such as a new model launch or a retirement.
- Assign the line to the team that decides what ships, and set alerts at thresholds you choose on the project budget.
- Review the forecast against actuals monthly. A run that cost twice its estimate points to a wrong token assumption, a retry storm or an unpriced model, and each of those is fixable.
For the finance reader, a short monthly view is enough: planned runs, completed runs, estimated spend against budget, and the largest run. The usage and runs endpoints and the FOCUS export carry the data, so eval spend can sit beside the rest of your cloud cost rather than in a separate spreadsheet.
A checklist before the next run
- The run has a purpose and an owner, and appears in the quarter's list of planned runs.
- The estimate was computed from the real dataset, variants and evaluators, and nothing is unpriced.
- Negotiated rates are entered, so the estimate is not on list rates by accident.
- The judge model was chosen by calibration, and its output length is bounded.
- Reasoning on or off is a recorded choice, with a pilot showing it does not change the answers.
- Variants and repeats are the minimum that answers the question.
- The environment is set explicitly and is not production.
- Every call carries a workflow name and a trace id, and the grouping tag is promoted.
- The per-run cap is at least the estimate, and the eval key and project have blocking budgets above it.
- Live judging has a monthly limit, and an alert fires if the limit is reached.
- After the run, actual cost from the runs list is compared with the estimate and the gap is explained.
Doing this on your own provider accounts
If your provider accounts are connected to INFRO, the pieces above are available together: experiments with an estimate and a spend cap, evaluators with a monthly limit, environments, attribution, runs and budgets. The model spend stays on your own provider accounts at your own rates; INFRO takes 0% of it, and the subscription pays for the control plane.5 You can build the same controls yourself with a wrapper around your provider SDK, but the estimate, the cap and the tagging then become code you maintain.
