Skip to content

Agents and multimodal systemsMethodology

Scoring agent trajectories, not just answers: a method

An agent can pass every final-answer check and still call the wrong tool, repeat itself or take twice the steps it needs. This method scores the path from traces, with deterministic checks first, and compares two models on the same set of tasks.

By INFRO Editorial

Published

9 min read

A glowing autonomous agent core coordinating tools, memory, and multiple AI model routes

In short

Score each run as a trajectory: tool choice against an allowed set, steps against a known minimum, repeated calls and retries, and estimated cost per run. Then replay the same task set on both models, repeat each task, and compare paired differences rather than pooled averages.

Key takeaways

  • A final-answer pass rate hides wrong tool calls, loops and wasted steps, so score the path as well as the result.
  • Score tool choice, repeats and step ratio with deterministic code first; use an LLM judge only for questions code cannot answer, and calibrate it.
  • Keep one trace id per run and one workflow name per arm, so each model's runs can be listed, costed and scored separately.
  • Compare models on the same tasks with repeats and paired differences, and drop runs where a failover changed the model mid-run.
  • Report cost per passing run next to step ratio, and remember that model cost excludes the tools' own charges.

Why the final answer is the wrong unit

A final-answer check asks one question: did the run end in an acceptable result? It cannot see that the agent called a search tool five times with the same query, picked a write tool where a read tool would do, or took eleven steps for a task that needs four. Those runs pass, cost more, run longer and fail more often under load, and none of it shows up in a pass rate.

The fix is to treat the trajectory as the thing under test: the ordered list of model calls and tool calls between the task and the answer. Platform vendors describe the same need. AWS says of AgentCore that you can see what steps an agent took, what it called and where it went off track, and test variations against real traffic.1 This article is a method you can run on your own traces, with any agent framework.

The method has four parts: capture a trajectory per run, score four things deterministically, store the scores where you can filter on them, and compare two models on the same task set with paired statistics. Cost per task and general tracing are covered elsewhere on this blog; here they are inputs, not the topic.

Capture one trajectory per run

A run needs a stable id and a step list. In INFRO, a run is the set of requests that carry the same trace_id for one workflow, so you send metadata.trace_id (or the X-INFRO-Trace-Id header) and metadata.workflow on every model call.2 Runs are grouped by workflow and trace id together, because trace ids are yours and two workflows can reuse one.2 Use that to your advantage: give each arm of a comparison its own workflow name, for example triage-eval-a and triage-eval-b, and the two arms never mix.

Tool calls need a second source. Tool names and arguments live in the model's responses, and prompt and completion logging is off by default with a retention window you choose.3 If you do not log payloads, emit the tool name, a normalized argument fingerprint, a success flag and the step index from your own agent loop, as span attributes or log lines keyed by the same trace id. Tools reached through INFRO's MCP gateway are recorded on the same trace as the agent's model calls, and INFRO never executes a tool itself.4

If you export spans to your own collector, check which GenAI conventions you target. OpenTelemetry's GenAI conventions list agent spans and MCP conventions, but the pages in the main repository now only point to a separate repository where they are maintained.5 Pin the version you instrument against, since names in a moving convention change.

The minimum record per step:

trajectory.py
from dataclasses import dataclass
import json

@dataclass
class Step:
    index: int
    kind: str            # "model" or "tool"
    tool: str | None     # tool name, None for model-only steps
    args: dict | None    # arguments as sent
    ok: bool             # tool returned without error
    cost: float          # estimated provider cost of the model call, USD

def fingerprint(step: Step) -> tuple:
    # Same tool + same arguments, key order ignored
    return (step.tool, json.dumps(step.args or {}, sort_keys=True))

Score four things from the trace

Keep the first pass deterministic. Code is repeatable, free of judge variance, and you can run it on every run rather than a sample. Each task in your reference set carries a few labels: the tools it needs, the tools it may use, the tools it must never use, and the smallest number of tool steps a good run takes.

ScoreQuestionComputation
tool_precisionWere the tools called the right ones?Tool steps whose tool is in the needed or allowed set, divided by all tool steps
tool_recallDid it use what the task needs?Needed tools called at least once, divided by needed tools
forbidden_callsDid it touch a tool it must not?Count of calls to the forbidden set
step_ratioWas the path short?Minimum known tool steps divided by actual tool steps, capped at 1
repeat_rateDid it repeat itself?Tool steps whose fingerprint already occurred, divided by tool steps
max_loopDid it get stuck?Longest run of consecutive identical fingerprints
run_costWhat did the run cost?Sum of estimated provider cost over its model calls
the per-run trajectory scores and how each is computed.

Retries need care because they are two different behaviours. A call that fails and is then reissued with changed arguments is a legitimate repair. A call that fails and is reissued unchanged is a repeat, and several in a row is a loop. Counting both as "retries" mixes a healthy behaviour with the one you want to catch, so the scorer separates them:

score_run.py
def score_run(steps, needed, allowed, forbidden, min_steps):
    tools = [s for s in steps if s.kind == "tool"]
    n = len(tools)
    if n == 0:
        return {"tool_precision": 0.0, "tool_recall": 0.0, "step_ratio": 0.0}

    ok_set = set(needed) | set(allowed)
    seen, repeats, streak, max_loop = set(), 0, 0, 1
    prev = None
    for s in tools:
        fp = fingerprint(s)
        if fp in seen:
            repeats += 1
        seen.add(fp)
        streak = streak + 1 if fp == prev else 1
        max_loop = max(max_loop, streak)
        prev = fp

    called = {s.tool for s in tools}
    return {
        "tool_precision": sum(s.tool in ok_set for s in tools) / n,
        "tool_recall": len(called & set(needed)) / max(len(needed), 1),
        "forbidden_calls": sum(s.tool in forbidden for s in tools),
        "step_ratio": min(1.0, min_steps / n),
        "repeat_rate": repeats / n,
        "max_loop": max_loop,
        "run_cost": sum(s.cost for s in steps if s.kind == "model"),
    }

Two judgement calls matter. First, min_steps should come from a trajectory a person has accepted, not from the best a model ever did, or the baseline moves with the model you are testing. Second, the tool catalogue is part of the test. MCP is an open standard for connecting AI applications to external tools and data,6 so one agent can see dozens of tools from several servers. A model that picks well from six tools may pick badly from sixty, and tool_precision is where that appears.

Only after these are in place add an LLM judge for what code cannot decide, such as whether a search was redundant given what the context already held. Calibrate it against human labels first; the article on judge calibration covers how.

Store scores where you can filter on them

Scores are only useful if you can find the worst runs. INFRO accepts a score on a request or on a run from your code: a number, a boolean or a category, with a name of 1 to 64 characters from a-z 0-9 _ . -, and up to 100 scores in one call.7 Scores do not count against the request allowance.7

push_scores.py
import os, requests

def push(trace_id, scores):
    body = {"scores": [
        {"trace_id": trace_id, "name": f"traj.{k}", "value": float(v)}
        for k, v in scores.items()
    ]}
    r = requests.post(
        "https://api.infro.io/v1/scores",
        headers={"Authorization": f"Bearer {os.environ['INFRO_API_KEY']}"},
        json=body, timeout=30,
    )
    r.raise_for_status()

One caveat shapes the design. The runs list filter score_below compares against the run's mean numeric score and leaves out runs with no numeric score.2 If you push max_loop = 7 beside step_ratio = 0.4, the mean means nothing. Either push only values on a 0 to 1 scale to the run, or push one composite and keep the raw counts in your own warehouse. Booleans such as forbidden_any are fine for filtering in other tools but do not enter that mean.

Then the list calls do the triage. GET /v1/runs sorts by costliest, slowest or errors, filters by workflow, model and min_cost, and each row carries requests, errors, duration, models, cost and whether the cost is partial.2

Shell
curl "https://api.infro.io/v1/runs?workflow=triage-eval-b&score_below=0.6&sort=costliest" \
  -H "Authorization: Bearer $INFRO_API_KEY"

For a warehouse join, the request log can be exported as CSV or pulled with GET /v1/requests, which also supports sort=costliest.3

Compare two models on the same trajectories

Pooled averages across different tasks mislead, because the arm that happened to draw harder tasks looks worse. The comparison has to be paired.

  1. Freeze the task set. Build a dataset of inputs with expected outputs and metadata. You can create it from logged requests in one click, upload JSONL or CSV, or send it from code, and a dataset keeps a version that changes whenever its items change.8 Put the needed, allowed and forbidden tools and min_steps in each item's metadata.
  2. Run both arms on every task, several times. Agent runs are not deterministic. Run each task at least three times per arm and average per task before comparing.
  3. Score every run with the same code. Same scorer version, same labels, same tool catalogue, same system prompt except where the prompt is the variable.
  4. Compute the per-task difference. For each task take arm B minus arm A on each score, then summarize the differences.
  5. Put an interval on it. A paired bootstrap over tasks is enough:
compare.py
import random, statistics

def paired_interval(diffs, n=5000, seed=7):
    rng = random.Random(seed)
    means = sorted(
        statistics.fmean(rng.choices(diffs, k=len(diffs))) for _ in range(n)
    )
    return means[int(0.025 * n)], means[int(0.975 * n)]

If the interval on a difference excludes zero, the difference is likely real for this task set. If it straddles zero, the sample cannot tell the arms apart on that score, which is a result too.

INFRO's experiments run a dataset through up to six variants, judged by up to five evaluators, and show the request count and estimated provider cost before starting, with a spend cap it stops at.9 The docs describe items as chat messages and variants as a model, prompt version and settings.9 That fits step-level replay: put the conversation up to step n in the item and compare which tool each model chooses next. A full multi-step loop with real tool execution is better driven by your own harness, one workflow name per arm, with scores pushed as above. I did not confirm that experiment variants carry tool definitions, so check the docs before relying on it for tool choice.

Read the result as a trade-off

Cost per run needs the same suspicion as the other scores. INFRO records what each request used as your provider reports it and estimates what your provider will bill, while your providers invoice you directly,10 so the run figure is an estimate, not an invoice. It covers model calls only: INFRO states that tool calls through its MCP gateway are not priced,4 so search, fetch or sandbox charges from the services behind your tools have to be added from their own bills. Check cost_is_partial and unpriced_requests on each run: a run with unpriced requests understates cost, and averaging it in as zero flatters the arm.2 If your contract rates differ from list rates, enter them first so the estimate uses them.

Report cost per passing run, not per run: total estimated cost divided by passing runs, so a model that is cheap per call but fails or loops more is charged for its failures. Also report the median and the 90th percentile; the tail is where loops live.

A workable gate before switching models: pass rate not lower beyond your tolerance, no new forbidden calls, max_loop no worse at the 95th percentile, and cost per passing run within a limit you set in advance. For production, pair the gate with step and spend limits so a loop is stopped rather than scored afterwards; the article on bounding agent loops covers that.

Questions readers ask

How many tasks does a trajectory set need?
Enough that the confidence interval on the paired difference is narrower than the difference you care about. Start with a set you can label by hand, compute the interval, and add tasks from the buckets where it is widest. A dataset holds up to 10,000 items, so the limit is your labelling effort, not storage.8
Can an LLM judge decide whether a step was necessary?
It can, but treat it as a second tier. Use code for tool choice, repeats and step ratio, and use a judge only for questions such as whether a search was redundant given the context. Calibrate it against a human-labelled sample before its scores gate anything.
Should failed tool calls count as wasted steps?
Count them separately. A call that failed and was then fixed with different arguments is a retry; a call that failed and was sent again unchanged is a repeat. The first can be healthy, the second is the signal for a loop.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Runs

    Available with a plan

    Requests that share a trace id listed as one run each: requests, errors, duration, models, estimated provider cost, scores and prompt versions, filtered and sorted over a window of up to 31 days. In the API, the console, the management API and MCP, on every plan.

  • Scores and end-user feedback

    Available with a plan

    Scores on a request or a run from your code, from eval monitors and from members in the console, and feedback from the people reading your answers: each response carries a signed token a browser can send with a thumbs up or down, without an API key. Scores do not count against the request allowance. On every plan.

  • Datasets

    Available with a plan

    Inputs and expected outputs built from logged requests in one click, uploaded as JSONL or CSV, or sent from code, kept until you delete them. On every plan.

  • Experiments

    Available with a plan

    A dataset run through the gateway across up to six variants (a model or route, a prompt version, settings), judged by your evaluators and compared side by side, with the request count and estimated provider cost shown before it starts and a spend cap it stops at. On every plan.

  • Request traces and payload logs

    Available with a plan

    Per-request timing detail, with prompt and completion logging off by default and a retention window you choose.

    Read the documentation for Request traces and payload logs
  • MCP gateway for your agents' tools

    Available with a plan

    Register your own MCP servers once with their credentials, approve and pin each tool's definition, grant gateway keys access, and let agents call those tools through INFRO with guardrails, a call log on the same trace as their model calls, and kill switches checked on every call. INFRO executes no tool.

    Read the documentation for MCP gateway for your agents' tools
  • OpenTelemetry export

    Available with a plan

    Spans pushed to your own collector from the request path, with the endpoint and headers set in the console. Included on every plan.

    Read the documentation for OpenTelemetry export
  • Usage, request and FOCUS exports

    Available with a plan

    GET /v1/usage and GET /v1/requests on any gateway key, the request log and usage series as CSV, plus a FOCUS v1.4 Cost and Usage dataset as JSON or CSV, so AI spend lands in your FinOps platform beside the rest of your cloud cost.

    Read the documentation for Usage, request and FOCUS exports

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Amazon Bedrock AgentCore (opens in a new tab) — Amazon Web Services. Accessed Nov 18, 2026.
  2. 2.Runs (opens in a new tab) — INFRO. Accessed Nov 18, 2026.
  3. 3.Request traces (opens in a new tab) — INFRO. Accessed Nov 18, 2026.
  4. 4.MCP gateway (opens in a new tab) — INFRO. Accessed Nov 18, 2026.
  5. 5.Generative AI semantic conventions (moved notice) (opens in a new tab) — OpenTelemetry. Accessed Nov 18, 2026.
  6. 6.What is the Model Context Protocol (MCP)? (opens in a new tab) — Model Context Protocol. Accessed Nov 18, 2026.
  7. 7.Scores and feedback (opens in a new tab) — INFRO. Accessed Nov 18, 2026.
  8. 8.Datasets (opens in a new tab) — INFRO. Accessed Nov 18, 2026.
  9. 9.Experiments (opens in a new tab) — INFRO. Accessed Nov 18, 2026.
  10. 10.INFRO documentation overview (opens in a new tab) — INFRO. Accessed Nov 18, 2026.

About this article

Read on 2026-11-18: the OpenTelemetry GenAI semantic-conventions page (a moved notice, so no attribute names are quoted), the MCP introduction, the AWS AgentCore overview, and INFRO's docs overview, runs, scores, experiments, datasets, request traces and MCP gateway pages. Left out as unconfirmed: span attribute names for agents, and whether experiment variants carry tool definitions.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.