Why the final answer is the wrong unit
A final-answer check asks one question: did the run end in an acceptable result? It cannot see that the agent called a search tool five times with the same query, picked a write tool where a read tool would do, or took eleven steps for a task that needs four. Those runs pass, cost more, run longer and fail more often under load, and none of it shows up in a pass rate.
The fix is to treat the trajectory as the thing under test: the ordered list of model calls and tool calls between the task and the answer. Platform vendors describe the same need. AWS says of AgentCore that you can see what steps an agent took, what it called and where it went off track, and test variations against real traffic.1 This article is a method you can run on your own traces, with any agent framework.
The method has four parts: capture a trajectory per run, score four things deterministically, store the scores where you can filter on them, and compare two models on the same task set with paired statistics. Cost per task and general tracing are covered elsewhere on this blog; here they are inputs, not the topic.
Capture one trajectory per run
A run needs a stable id and a step list. In INFRO, a run is the set of requests that carry the same trace_id for one workflow, so you send metadata.trace_id (or the X-INFRO-Trace-Id header) and metadata.workflow on every model call.2 Runs are grouped by workflow and trace id together, because trace ids are yours and two workflows can reuse one.2 Use that to your advantage: give each arm of a comparison its own workflow name, for example triage-eval-a and triage-eval-b, and the two arms never mix.
Tool calls need a second source. Tool names and arguments live in the model's responses, and prompt and completion logging is off by default with a retention window you choose.3 If you do not log payloads, emit the tool name, a normalized argument fingerprint, a success flag and the step index from your own agent loop, as span attributes or log lines keyed by the same trace id. Tools reached through INFRO's MCP gateway are recorded on the same trace as the agent's model calls, and INFRO never executes a tool itself.4
If you export spans to your own collector, check which GenAI conventions you target. OpenTelemetry's GenAI conventions list agent spans and MCP conventions, but the pages in the main repository now only point to a separate repository where they are maintained.5 Pin the version you instrument against, since names in a moving convention change.
The minimum record per step:
from dataclasses import dataclass
import json
@dataclass
class Step:
index: int
kind: str # "model" or "tool"
tool: str | None # tool name, None for model-only steps
args: dict | None # arguments as sent
ok: bool # tool returned without error
cost: float # estimated provider cost of the model call, USD
def fingerprint(step: Step) -> tuple:
# Same tool + same arguments, key order ignored
return (step.tool, json.dumps(step.args or {}, sort_keys=True))Score four things from the trace
Keep the first pass deterministic. Code is repeatable, free of judge variance, and you can run it on every run rather than a sample. Each task in your reference set carries a few labels: the tools it needs, the tools it may use, the tools it must never use, and the smallest number of tool steps a good run takes.
| Score | Question | Computation |
|---|---|---|
tool_precision | Were the tools called the right ones? | Tool steps whose tool is in the needed or allowed set, divided by all tool steps |
tool_recall | Did it use what the task needs? | Needed tools called at least once, divided by needed tools |
forbidden_calls | Did it touch a tool it must not? | Count of calls to the forbidden set |
step_ratio | Was the path short? | Minimum known tool steps divided by actual tool steps, capped at 1 |
repeat_rate | Did it repeat itself? | Tool steps whose fingerprint already occurred, divided by tool steps |
max_loop | Did it get stuck? | Longest run of consecutive identical fingerprints |
run_cost | What did the run cost? | Sum of estimated provider cost over its model calls |
Retries need care because they are two different behaviours. A call that fails and is then reissued with changed arguments is a legitimate repair. A call that fails and is reissued unchanged is a repeat, and several in a row is a loop. Counting both as "retries" mixes a healthy behaviour with the one you want to catch, so the scorer separates them:
def score_run(steps, needed, allowed, forbidden, min_steps):
tools = [s for s in steps if s.kind == "tool"]
n = len(tools)
if n == 0:
return {"tool_precision": 0.0, "tool_recall": 0.0, "step_ratio": 0.0}
ok_set = set(needed) | set(allowed)
seen, repeats, streak, max_loop = set(), 0, 0, 1
prev = None
for s in tools:
fp = fingerprint(s)
if fp in seen:
repeats += 1
seen.add(fp)
streak = streak + 1 if fp == prev else 1
max_loop = max(max_loop, streak)
prev = fp
called = {s.tool for s in tools}
return {
"tool_precision": sum(s.tool in ok_set for s in tools) / n,
"tool_recall": len(called & set(needed)) / max(len(needed), 1),
"forbidden_calls": sum(s.tool in forbidden for s in tools),
"step_ratio": min(1.0, min_steps / n),
"repeat_rate": repeats / n,
"max_loop": max_loop,
"run_cost": sum(s.cost for s in steps if s.kind == "model"),
}Two judgement calls matter. First, min_steps should come from a trajectory a person has accepted, not from the best a model ever did, or the baseline moves with the model you are testing. Second, the tool catalogue is part of the test. MCP is an open standard for connecting AI applications to external tools and data,6 so one agent can see dozens of tools from several servers. A model that picks well from six tools may pick badly from sixty, and tool_precision is where that appears.
Only after these are in place add an LLM judge for what code cannot decide, such as whether a search was redundant given what the context already held. Calibrate it against human labels first; the article on judge calibration covers how.
Store scores where you can filter on them
Scores are only useful if you can find the worst runs. INFRO accepts a score on a request or on a run from your code: a number, a boolean or a category, with a name of 1 to 64 characters from a-z 0-9 _ . -, and up to 100 scores in one call.7 Scores do not count against the request allowance.7
import os, requests
def push(trace_id, scores):
body = {"scores": [
{"trace_id": trace_id, "name": f"traj.{k}", "value": float(v)}
for k, v in scores.items()
]}
r = requests.post(
"https://api.infro.io/v1/scores",
headers={"Authorization": f"Bearer {os.environ['INFRO_API_KEY']}"},
json=body, timeout=30,
)
r.raise_for_status()One caveat shapes the design. The runs list filter score_below compares against the run's mean numeric score and leaves out runs with no numeric score.2 If you push max_loop = 7 beside step_ratio = 0.4, the mean means nothing. Either push only values on a 0 to 1 scale to the run, or push one composite and keep the raw counts in your own warehouse. Booleans such as forbidden_any are fine for filtering in other tools but do not enter that mean.
Then the list calls do the triage. GET /v1/runs sorts by costliest, slowest or errors, filters by workflow, model and min_cost, and each row carries requests, errors, duration, models, cost and whether the cost is partial.2
curl "https://api.infro.io/v1/runs?workflow=triage-eval-b&score_below=0.6&sort=costliest" \
-H "Authorization: Bearer $INFRO_API_KEY"For a warehouse join, the request log can be exported as CSV or pulled with GET /v1/requests, which also supports sort=costliest.3
Compare two models on the same trajectories
Pooled averages across different tasks mislead, because the arm that happened to draw harder tasks looks worse. The comparison has to be paired.
- Freeze the task set. Build a dataset of inputs with expected outputs and metadata. You can create it from logged requests in one click, upload JSONL or CSV, or send it from code, and a dataset keeps a version that changes whenever its items change.8 Put the needed, allowed and forbidden tools and
min_stepsin each item's metadata. - Run both arms on every task, several times. Agent runs are not deterministic. Run each task at least three times per arm and average per task before comparing.
- Score every run with the same code. Same scorer version, same labels, same tool catalogue, same system prompt except where the prompt is the variable.
- Compute the per-task difference. For each task take arm B minus arm A on each score, then summarize the differences.
- Put an interval on it. A paired bootstrap over tasks is enough:
import random, statistics
def paired_interval(diffs, n=5000, seed=7):
rng = random.Random(seed)
means = sorted(
statistics.fmean(rng.choices(diffs, k=len(diffs))) for _ in range(n)
)
return means[int(0.025 * n)], means[int(0.975 * n)]If the interval on a difference excludes zero, the difference is likely real for this task set. If it straddles zero, the sample cannot tell the arms apart on that score, which is a result too.
INFRO's experiments run a dataset through up to six variants, judged by up to five evaluators, and show the request count and estimated provider cost before starting, with a spend cap it stops at.9 The docs describe items as chat messages and variants as a model, prompt version and settings.9 That fits step-level replay: put the conversation up to step n in the item and compare which tool each model chooses next. A full multi-step loop with real tool execution is better driven by your own harness, one workflow name per arm, with scores pushed as above. I did not confirm that experiment variants carry tool definitions, so check the docs before relying on it for tool choice.
Read the result as a trade-off
Cost per run needs the same suspicion as the other scores. INFRO records what each request used as your provider reports it and estimates what your provider will bill, while your providers invoice you directly,10 so the run figure is an estimate, not an invoice. It covers model calls only: INFRO states that tool calls through its MCP gateway are not priced,4 so search, fetch or sandbox charges from the services behind your tools have to be added from their own bills. Check cost_is_partial and unpriced_requests on each run: a run with unpriced requests understates cost, and averaging it in as zero flatters the arm.2 If your contract rates differ from list rates, enter them first so the estimate uses them.
Report cost per passing run, not per run: total estimated cost divided by passing runs, so a model that is cheap per call but fails or loops more is charged for its failures. Also report the median and the 90th percentile; the tail is where loops live.
A workable gate before switching models: pass rate not lower beyond your tolerance, no new forbidden calls, max_loop no worse at the 95th percentile, and cost per passing run within a limit you set in advance. For production, pair the gate with step and spend limits so a loop is stopped rather than scored afterwards; the article on bounding agent loops covers that.
