# Scoring agent trajectories, not just answers: a method

URL: https://infro.io/blog/scoring-agent-trajectories-not-just-answers-a-method
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-18
Last modified: 2026-11-18
Last reviewed: 2026-11-18 (next review due 2027-05-17)
Topic: Agents and multimodal systems
Format: Methodology

> Score each run as a trajectory: tool choice against an allowed set, steps against a known minimum, repeated calls and retries, and estimated cost per run. Then replay the same task set on both models, repeat each task, and compare paired differences rather than pooled averages.

Key takeaways:
- A final-answer pass rate hides wrong tool calls, loops and wasted steps, so score the path as well as the result.
- Score tool choice, repeats and step ratio with deterministic code first; use an LLM judge only for questions code cannot answer, and calibrate it.
- Keep one trace id per run and one workflow name per arm, so each model's runs can be listed, costed and scored separately.
- Compare models on the same tasks with repeats and paired differences, and drop runs where a failover changed the model mid-run.
- Report cost per passing run next to step ratio, and remember that model cost excludes the tools' own charges.

## Why the final answer is the wrong unit

A final-answer check asks one question: did the run end in an acceptable result? It cannot see that the agent called a search tool five times with the same query, picked a write tool where a read tool would do, or took eleven steps for a task that needs four. Those runs pass, cost more, run longer and fail more often under load, and none of it shows up in a pass rate.

The fix is to treat the **trajectory** as the thing under test: the ordered list of model calls and tool calls between the task and the answer. Platform vendors describe the same need. AWS says of AgentCore that you can see what steps an agent took, what it called and where it went off track, and test variations against real traffic.[1] This article is a method you can run on your own traces, with any agent framework.

The method has four parts: capture a trajectory per run, score four things deterministically, store the scores where you can filter on them, and compare two models on the same task set with paired statistics. Cost per task and general tracing are covered elsewhere on this blog; here they are inputs, not the topic.

## Capture one trajectory per run

A run needs a stable id and a step list. In INFRO, a run is the set of requests that carry the same `trace_id` for one workflow, so you send `metadata.trace_id` (or the `X-INFRO-Trace-Id` header) and `metadata.workflow` on every model call.[2] Runs are grouped by workflow and trace id together, because trace ids are yours and two workflows can reuse one.[2] Use that to your advantage: give each arm of a comparison its own workflow name, for example `triage-eval-a` and `triage-eval-b`, and the two arms never mix.

Tool calls need a second source. Tool names and arguments live in the model's responses, and prompt and completion logging is off by default with a retention window you choose.[3] If you do not log payloads, emit the tool name, a normalized argument fingerprint, a success flag and the step index from your own agent loop, as span attributes or log lines keyed by the same trace id. Tools reached through INFRO's MCP gateway are recorded on the same trace as the agent's model calls, and INFRO never executes a tool itself.[4]

If you export spans to your own collector, check which GenAI conventions you target. OpenTelemetry's GenAI conventions list agent spans and MCP conventions, but the pages in the main repository now only point to a separate repository where they are maintained.[5] Pin the version you instrument against, since names in a moving convention change.

The minimum record per step:

```python
from dataclasses import dataclass
import json

@dataclass
class Step:
    index: int
    kind: str            # "model" or "tool"
    tool: str | None     # tool name, None for model-only steps
    args: dict | None    # arguments as sent
    ok: bool             # tool returned without error
    cost: float          # estimated provider cost of the model call, USD

def fingerprint(step: Step) -> tuple:
    # Same tool + same arguments, key order ignored
    return (step.tool, json.dumps(step.args or {}, sort_keys=True))
```

## Score four things from the trace

Keep the first pass deterministic. Code is repeatable, free of judge variance, and you can run it on every run rather than a sample. Each task in your reference set carries a few labels: the tools it needs, the tools it may use, the tools it must never use, and the smallest number of tool steps a good run takes.

| Score | Question | Computation |
| --- | --- | --- |
| `tool_precision` | Were the tools called the right ones? | Tool steps whose tool is in the needed or allowed set, divided by all tool steps |
| `tool_recall` | Did it use what the task needs? | Needed tools called at least once, divided by needed tools |
| `forbidden_calls` | Did it touch a tool it must not? | Count of calls to the forbidden set |
| `step_ratio` | Was the path short? | Minimum known tool steps divided by actual tool steps, capped at 1 |
| `repeat_rate` | Did it repeat itself? | Tool steps whose fingerprint already occurred, divided by tool steps |
| `max_loop` | Did it get stuck? | Longest run of consecutive identical fingerprints |
| `run_cost` | What did the run cost? | Sum of estimated provider cost over its model calls |

Table: the per-run trajectory scores and how each is computed.

Retries need care because they are two different behaviours. A call that fails and is then reissued with changed arguments is a legitimate repair. A call that fails and is reissued unchanged is a repeat, and several in a row is a loop. Counting both as "retries" mixes a healthy behaviour with the one you want to catch, so the scorer separates them:

```python
def score_run(steps, needed, allowed, forbidden, min_steps):
    tools = [s for s in steps if s.kind == "tool"]
    n = len(tools)
    if n == 0:
        return {"tool_precision": 0.0, "tool_recall": 0.0, "step_ratio": 0.0}

    ok_set = set(needed) | set(allowed)
    seen, repeats, streak, max_loop = set(), 0, 0, 1
    prev = None
    for s in tools:
        fp = fingerprint(s)
        if fp in seen:
            repeats += 1
        seen.add(fp)
        streak = streak + 1 if fp == prev else 1
        max_loop = max(max_loop, streak)
        prev = fp

    called = {s.tool for s in tools}
    return {
        "tool_precision": sum(s.tool in ok_set for s in tools) / n,
        "tool_recall": len(called & set(needed)) / max(len(needed), 1),
        "forbidden_calls": sum(s.tool in forbidden for s in tools),
        "step_ratio": min(1.0, min_steps / n),
        "repeat_rate": repeats / n,
        "max_loop": max_loop,
        "run_cost": sum(s.cost for s in steps if s.kind == "model"),
    }
```

Two judgement calls matter. First, `min_steps` should come from a trajectory a person has accepted, not from the best a model ever did, or the baseline moves with the model you are testing. Second, the tool catalogue is part of the test. MCP is an open standard for connecting AI applications to external tools and data,[6] so one agent can see dozens of tools from several servers. A model that picks well from six tools may pick badly from sixty, and `tool_precision` is where that appears.

Only after these are in place add an LLM judge for what code cannot decide, such as whether a search was redundant given what the context already held. Calibrate it against human labels first; the article on judge calibration covers how.

## Store scores where you can filter on them

Scores are only useful if you can find the worst runs. INFRO accepts a score on a request or on a run from your code: a number, a boolean or a category, with a name of 1 to 64 characters from `a-z 0-9 _ . -`, and up to 100 scores in one call.[7] Scores do not count against the request allowance.[7]

```python
import os, requests

def push(trace_id, scores):
    body = {"scores": [
        {"trace_id": trace_id, "name": f"traj.{k}", "value": float(v)}
        for k, v in scores.items()
    ]}
    r = requests.post(
        "https://api.infro.io/v1/scores",
        headers={"Authorization": f"Bearer {os.environ['INFRO_API_KEY']}"},
        json=body, timeout=30,
    )
    r.raise_for_status()
```

One caveat shapes the design. The runs list filter `score_below` compares against the run's *mean* numeric score and leaves out runs with no numeric score.[2] If you push `max_loop = 7` beside `step_ratio = 0.4`, the mean means nothing. Either push only values on a 0 to 1 scale to the run, or push one composite and keep the raw counts in your own warehouse. Booleans such as `forbidden_any` are fine for filtering in other tools but do not enter that mean.

Then the list calls do the triage. `GET /v1/runs` sorts by `costliest`, `slowest` or `errors`, filters by `workflow`, `model` and `min_cost`, and each row carries requests, errors, duration, models, cost and whether the cost is partial.[2]

```bash
curl "https://api.infro.io/v1/runs?workflow=triage-eval-b&score_below=0.6&sort=costliest" \
  -H "Authorization: Bearer $INFRO_API_KEY"
```

For a warehouse join, the request log can be exported as CSV or pulled with `GET /v1/requests`, which also supports `sort=costliest`.[3]

## Compare two models on the same trajectories

Pooled averages across different tasks mislead, because the arm that happened to draw harder tasks looks worse. The comparison has to be paired.

1. **Freeze the task set.** Build a dataset of inputs with expected outputs and metadata. You can create it from logged requests in one click, upload JSONL or CSV, or send it from code, and a dataset keeps a version that changes whenever its items change.[8] Put the needed, allowed and forbidden tools and `min_steps` in each item's metadata.
2. **Run both arms on every task, several times.** Agent runs are not deterministic. Run each task at least three times per arm and average per task before comparing.
3. **Score every run with the same code.** Same scorer version, same labels, same tool catalogue, same system prompt except where the prompt is the variable.
4. **Compute the per-task difference.** For each task take arm B minus arm A on each score, then summarize the differences.
5. **Put an interval on it.** A paired bootstrap over tasks is enough:

```python
import random, statistics

def paired_interval(diffs, n=5000, seed=7):
    rng = random.Random(seed)
    means = sorted(
        statistics.fmean(rng.choices(diffs, k=len(diffs))) for _ in range(n)
    )
    return means[int(0.025 * n)], means[int(0.975 * n)]
```

If the interval on a difference excludes zero, the difference is likely real for this task set. If it straddles zero, the sample cannot tell the arms apart on that score, which is a result too.

INFRO's experiments run a dataset through up to six variants, judged by up to five evaluators, and show the request count and estimated provider cost before starting, with a spend cap it stops at.[9] The docs describe items as chat messages and variants as a model, prompt version and settings.[9] That fits step-level replay: put the conversation up to step *n* in the item and compare which tool each model chooses next. A full multi-step loop with real tool execution is better driven by your own harness, one workflow name per arm, with scores pushed as above. I did not confirm that experiment variants carry tool definitions, so check the docs before relying on it for tool choice.

> **Warning: Exclude runs where the model changed mid-run**
> If a route fails over during a run, part of the trajectory belongs to a different model. The runs list reports the models each run used,[2] so drop any run with more than one model from both arms, or the comparison measures your failover policy instead of the models. See the article on failover mid-agent-run for what happens to state.

## Read the result as a trade-off

Cost per run needs the same suspicion as the other scores. INFRO records what each request used as your provider reports it and estimates what your provider will bill, while your providers invoice you directly,[10] so the run figure is an estimate, not an invoice. It covers model calls only: INFRO states that tool calls through its MCP gateway are not priced,[4] so search, fetch or sandbox charges from the services behind your tools have to be added from their own bills. Check `cost_is_partial` and `unpriced_requests` on each run: a run with unpriced requests understates cost, and averaging it in as zero flatters the arm.[2] If your contract rates differ from list rates, enter them first so the estimate uses them.

Report cost per *passing* run, not per run: total estimated cost divided by passing runs, so a model that is cheap per call but fails or loops more is charged for its failures. Also report the median and the 90th percentile; the tail is where loops live.

> **Example: Illustrative comparison (made-up figures)**
> Suppose 120 tasks, 3 runs per task per arm, scored as above. These numbers are invented to show how to read the table.
> 
> | Metric | Model A | Model B | Paired difference (B − A) |
> | --- | --- | --- | --- |
> | Pass rate | 0.88 | 0.89 | +0.01, interval spans zero |
> | Median step ratio | 0.80 | 0.62 | −0.18, interval excludes zero |
> | Repeat rate | 0.04 | 0.11 | +0.07, interval excludes zero |
> | p90 cost per run | $0.09 | $0.21 | +$0.12 |
> 
> Both models pass about the same share of tasks, so a final-answer check would call them equal. The trajectory scores show B takes longer paths and repeats calls, and its tail cost is more than double. The decision is then about whether B's other properties justify that.

A workable gate before switching models: pass rate not lower beyond your tolerance, no new forbidden calls, `max_loop` no worse at the 95th percentile, and cost per passing run within a limit you set in advance. For production, pair the gate with step and spend limits so a loop is stopped rather than scored afterwards; the article on bounding agent loops covers that.

## Questions

### How many tasks does a trajectory set need?

Enough that the confidence interval on the paired difference is narrower than the difference you care about. Start with a set you can label by hand, compute the interval, and add tasks from the buckets where it is widest. A dataset holds up to 10,000 items, so the limit is your labelling effort, not storage.[8]

### Can an LLM judge decide whether a step was necessary?

It can, but treat it as a second tier. Use code for tool choice, repeats and step ratio, and use a judge only for questions such as whether a search was redundant given the context. Calibrate it against a human-labelled sample before its scores gate anything.

### Should failed tool calls count as wasted steps?

Count them separately. A call that failed and was then fixed with different arguments is a retry; a call that failed and was sent again unchanged is a repeat. The first can be healthy, the second is the signal for a loop.

## Sources

1. Amazon Bedrock AgentCore — Amazon Web Services. https://aws.amazon.com/bedrock/agentcore/ (accessed 2026-11-18)
2. Runs — INFRO. https://infro.io/docs/observability/runs (accessed 2026-11-18)
3. Request traces — INFRO. https://infro.io/docs/observability/traces (accessed 2026-11-18)
4. MCP gateway — INFRO. https://infro.io/docs/mcp-gateway (accessed 2026-11-18)
5. Generative AI semantic conventions (moved notice) — OpenTelemetry. https://opentelemetry.io/docs/specs/semconv/gen-ai/ (accessed 2026-11-18)
6. What is the Model Context Protocol (MCP)? — Model Context Protocol. https://modelcontextprotocol.io/docs/getting-started/intro (accessed 2026-11-18)
7. Scores and feedback — INFRO. https://infro.io/docs/observability/scores (accessed 2026-11-18)
8. Datasets — INFRO. https://infro.io/docs/observability/datasets (accessed 2026-11-18)
9. Experiments — INFRO. https://infro.io/docs/observability/experiments (accessed 2026-11-18)
10. INFRO documentation overview — INFRO. https://infro.io/docs (accessed 2026-11-18)
