Tracing an AI agent run across many model calls

A failed agent run rarely fails on one line. It fails on step 14, after a tool returned a truncated page on step 9, which the model summarised wrongly on step 11, which made step 12 retry a search three times. Looking at step 14 alone tells you nothing. You need the run: every model call and tool call in order, with inputs, outcomes, timings and cost.

Anthropic's guidance on agents names the problem directly: autonomy brings "the potential for compounding errors"¹. Tracing is how you find where the compounding started.

What a run trace has to answer

Before choosing tools, write down the questions. A useful run trace answers these without anyone reading raw logs:

• Which calls belong to this run, in what order, and which one failed first?
• What did each step cost, and which step dominates cost across all runs of a workflow?
• Which model and which provider account served each call, and were there failover attempts?
• How long did the run spend in model calls versus tools, retrieval and queues?
• Did the run end on an answer, a limit, or an error?

Each question needs a join key. The run needs one identity, and each call needs a role within it.

One identity per run: W3C Trace Context

W3C Trace Context is a Recommendation that standardises how that identity travels between services. The traceparent header has four dash-separated fields — version, a 32-hex-character trace-id, a 16-hex-character parent-id, and trace-flags — and an all-zero trace-id or parent-id is invalid². When a receiver cannot parse the header, the specification says to start a new trace rather than propagate a broken one². That second rule matters for anything in the request path: a malformed header written by a library three hops away must never fail a user's request.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                └ flags (sampled)
             │  │                                └ parent-id: the caller's current span
             │  └ trace-id: the whole run
             └ version

If you already run OpenTelemetry, the trace-id your tracer generates for the run's root span is the run identity. Propagating it on every outbound call — to your model gateway, your vector database, your own services — is what makes them appear in one trace.

Name the spans the way the conventions do

OpenTelemetry publishes semantic conventions for generative AI, now maintained in their own repository; the old location on opentelemetry.io carries a notice that the content has moved³. The conventions define an inference span named {gen_ai.operation.name} {gen_ai.request.model}, with attributes such as gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens⁴. For agents they add an invoke_agent span, named invoke_agent {gen_ai.agent.name} when the name is known, and an execute-tool span carrying gen_ai.tool.name and gen_ai.tool.call.id⁵.

Every one of these is marked Development stability⁴⁵. They are the right names to use, and they can still change. Pin the names you depend on in dashboards and alerts, and re-check them when you upgrade instrumentation libraries.

Instrumenting the run in TypeScript

The pattern: open a root span for the run, open a child span for each model call and each tool call, inject the current context into outbound headers, and attach the same identity as request metadata so the gateway can group calls even where OpenTelemetry is not configured.

import { context, propagation, trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("support-agent");
const GATEWAY = "https://api.infro.io/v1/chat/completions";

type Step = "plan" | "tool_select" | "answer" | "critique"; // a fixed, small vocabulary

async function modelCall(runId: string, step: Step, body: Record<string, unknown>) {
  return tracer.startActiveSpan(`chat ${String(body.model)}`, async (span) => {
    span.setAttribute("gen_ai.operation.name", "chat");
    span.setAttribute("gen_ai.request.model", String(body.model));
    span.setAttribute("app.step", step);

    const headers: Record<string, string> = {
      Authorization: `Bearer ${process.env.INFRO_API_KEY}`,
      "Content-Type": "application/json",
    };
    propagation.inject(context.active(), headers); // adds traceparent (and tracestate)

    try {
      const res = await fetch(GATEWAY, {
        method: "POST",
        headers,
        body: JSON.stringify({
          ...body,
          metadata: { trace_id: runId, workflow: "support-agent", step },
        }),
      });
      const requestId = res.headers.get("X-INFRO-Request-Id");
      if (requestId) span.setAttribute("infro.request_id", requestId);
      if (!res.ok) throw new Error(`gateway ${res.status}: ${await res.text()}`);

      const json = await res.json();
      span.setAttribute("gen_ai.usage.input_tokens", json.usage?.prompt_tokens ?? 0);
      span.setAttribute("gen_ai.usage.output_tokens", json.usage?.completion_tokens ?? 0);
      return json;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

async function toolCall<T>(name: string, callId: string, fn: () => Promise<T>): Promise<T> {
  return tracer.startActiveSpan(`execute_tool ${name}`, async (span) => {
    span.setAttribute("gen_ai.operation.name", "execute_tool");
    span.setAttribute("gen_ai.tool.name", name);
    span.setAttribute("gen_ai.tool.call.id", callId);
    try {
      return await fn();
    } finally {
      span.end();
    }
  });
}

export async function runAgent(task: string) {
  return tracer.startActiveSpan("invoke_agent support-agent", async (root) => {
    // The OTel trace id doubles as the run id, so both systems agree.
    const runId = root.spanContext().traceId;
    root.setAttribute("gen_ai.agent.name", "support-agent");
    try {
      const plan = await modelCall(runId, "plan", {
        model: "anthropic/claude-sonnet-5",
        messages: [{ role: "user", content: task }],
        max_tokens: 800,
      });
      // ... toolCall(...) and further modelCall(...) steps ...
      return { runId, plan };
    } finally {
      root.end();
    }
  });
}

Two decisions in that code are worth copying. The step vocabulary is a closed type: free-text step names drift (answer, final_answer, Answer) and split one step across three groups. And the gateway's request id is recorded on the span, so an engineer looking at a slow span can open the exact gateway record behind it.

Reading a run in INFRO

INFRO records every model call routed through it with the model, the provider account that served it, fallback attempts, timings, token usage and an estimate of what your provider will bill for it. Four reserved metadata keys turn those records into runs (request traces):

A malformed reserved key is refused with 400 invalid_attribution naming the key, rather than silently dropped — a refusal on the first request of an integration is easier to fix than a hole in a cost report weeks later. Requests that cannot carry a JSON body can send X-INFRO-Trace-Id, X-INFRO-Workflow, X-INFRO-Step and X-INFRO-Feature headers instead.

Precedence is body metadata, then headers, then traceparent. In the gateway source, a malformed traceparent is treated as absent rather than refused, which matches the W3C rule above. One consequence is easy to miss: a metadata.trace_id sent without a traceparent groups requests in INFRO but does not become an OpenTelemetry trace id, because inventing trace context from an arbitrary string could join unrelated traces in your backend. Send traceparent if you want INFRO's spans inside your own trace.

GET /v1/traces/{trace_id} returns every request in one run, in order, with totals:

{
  "trace_id": "run_8f2c",
  "workflows": ["support-agent"],
  "started_at": "2026-09-29T09:14:02.118Z",
  "ended_at": "2026-09-29T09:14:31.904Z",
  "requests": [{ "id": "req_…", "step": "plan", "model": "anthropic/claude-sonnet-5", "…": "…" }],
  "truncated": false,
  "totals": {
    "requests": 9,
    "failed": 1,
    "estimated_cost": 0.0841,
    "unpriced_requests": 0,
    "cost_is_partial": false,
    "by_step": [{ "step": "answer", "requests": 3, "estimated_cost": 0.051, "share": 0.61 }],
    "by_modality": [{ "modality": "text", "requests": 9, "estimated_cost": 0.0841, "share": 1 }],
    "providers": ["anthropic"]
  },
  "covers": "model_calls_through_infro"
}

Read the last field first. The total covers model calls made through INFRO and nothing else: your vector database, your retrieval and your own compute are not in it (request traces). workflows is a list because trace ids are yours to generate and nothing stops two workflows sharing one by accident. When any successful call in the run had no price, cost_is_partial is true and the figure is a lower bound, never a filled-in guess. In the current gateway source, one answer returns up to 500 requests of a run and sets truncated beyond that. The same records are searchable in the console with trace:<id>, and GET /v1/requests/export?trace_id=… downloads one run as CSV without prompts or responses.

For cost across many runs, GET /v1/usage?group_by=step (or workflow, feature) aggregates the same estimates; a row per run is what the trace endpoint is for.

Sending spans to your own backend

On the Team plan and above, INFRO can deliver OTLP/HTTP traces to your collector's /v1/traces endpoint, with service name infro-gateway (exports). Spans carry operational metadata — request id, model, route, timing, token counts, estimated cost — plus the attribution values as infro.trace_id, infro.workflow, infro.step and infro.feature, and never captured prompts or responses. In the gateway source they also carry gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, and gen_ai.provider.name only for provider kinds whose conventional value is known.

Delivery is best effort and outside the response path: one retry on a server or network error, none on a 4xx, and dropped spans rather than blocked requests when the collector keeps failing. Treat the exported spans as observability, not as the accounting record; the request API and CSV export are the record.

Common tracing mistakes

• A new trace id per model call. The run disappears into unrelated fragments. Generate the id once, at the root.
• Reusing a request id as the run id. Frameworks that recycle request ids merge unrelated runs. Use a random, opaque id.
• Step names with parameters in them. search_customer_123 creates a group per customer. Put parameters in span attributes, not in the step name.
• Personal data in metadata. Trace ids and step names end up in URLs, exports and dashboards. Keep them opaque; send user identifiers as opaque ids.
• Presenting a gateway total as the run's cost. It is the model-call estimate. Retrieval, tools and compute are separate lines.

Checklist

• Each run has one trace id, created at the root and never regenerated by retries or sub-agents.
• Every outbound model call carries traceparent and the run id in metadata.
• Step names come from a fixed vocabulary per workflow.
• Tool calls, retrieval and queue waits have their own spans.
• Each model-call span records the gateway request id.
• Dashboards show cost and latency by step, and label run totals with what they cover.
• The GenAI attribute names your alerts depend on are pinned and re-checked on upgrade.