# Tracing an AI agent run across many model calls

URL: https://infro.io/blog/trace-ai-agent-runs
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Agents and multimodal systems
Format: Guide

> Give every agent run one trace id, propagate it on every model call with a W3C traceparent header, name each call's role as a step, and emit your own spans for tools and retrieval. Then a run can be read as one ordered timeline with cost by step — provided the total says which calls it covers and whether any were unpriced.

Key takeaways:
- A run needs one identity that every model call, tool call and retry carries; generate it once at the top of the run.
- W3C traceparent is the interoperable way to carry it, and a malformed one must be ignored rather than rejected.
- Name each call's role in the run as a step, with a small fixed vocabulary, so cost and latency can be grouped by step.
- A gateway sees model calls only. Emit your own spans for tools, retrieval and queue waits, and never present a gateway's run total as the whole cost.
- OpenTelemetry's GenAI conventions are still in development; pin the attribute names you depend on and re-check them on upgrade.

A failed agent run rarely fails on one line. It fails on step 14, after a tool returned a truncated page on step 9, which the model summarised wrongly on step 11, which made step 12 retry a search three times. Looking at step 14 alone tells you nothing. You need the run: every model call and tool call in order, with inputs, outcomes, timings and cost.

Anthropic's guidance on agents names the problem directly: autonomy brings "the potential for compounding errors"[1]. Tracing is how you find where the compounding started.

## What a run trace has to answer

Before choosing tools, write down the questions. A useful run trace answers these without anyone reading raw logs:

- Which calls belong to this run, in what order, and which one failed first?
- What did each step cost, and which step dominates cost across all runs of a workflow?
- Which model and which provider account served each call, and were there failover attempts?
- How long did the run spend in model calls versus tools, retrieval and queues?
- Did the run end on an answer, a limit, or an error?

Each question needs a join key. The run needs one identity, and each call needs a role within it.

## One identity per run: W3C Trace Context

W3C Trace Context is a Recommendation that standardises how that identity travels between services. The `traceparent` header has four dash-separated fields — version, a 32-hex-character trace-id, a 16-hex-character parent-id, and trace-flags — and an all-zero trace-id or parent-id is invalid[2]. When a receiver cannot parse the header, the specification says to start a new trace rather than propagate a broken one[2]. That second rule matters for anything in the request path: a malformed header written by a library three hops away must never fail a user's request.

```text
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                └ flags (sampled)
             │  │                                └ parent-id: the caller's current span
             │  └ trace-id: the whole run
             └ version
```

If you already run OpenTelemetry, the trace-id your tracer generates for the run's root span *is* the run identity. Propagating it on every outbound call — to your model gateway, your vector database, your own services — is what makes them appear in one trace.

## Name the spans the way the conventions do

OpenTelemetry publishes semantic conventions for generative AI, now maintained in their own repository; the old location on opentelemetry.io carries a notice that the content has moved[3]. The conventions define an inference span named `{gen_ai.operation.name} {gen_ai.request.model}`, with attributes such as `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`[4]. For agents they add an `invoke_agent` span, named `invoke_agent {gen_ai.agent.name}` when the name is known, and an execute-tool span carrying `gen_ai.tool.name` and `gen_ai.tool.call.id`[5].

Every one of these is marked **Development** stability[4][5]. They are the right names to use, and they can still change. Pin the names you depend on in dashboards and alerts, and re-check them when you upgrade instrumentation libraries.

[Diagram: One run as a span tree. The root span is yours; model calls are children; tool and retrieval spans are yours to emit, because no model gateway ever sees them.]

## Instrumenting the run in TypeScript

The pattern: open a root span for the run, open a child span for each model call and each tool call, inject the current context into outbound headers, and attach the same identity as request metadata so the gateway can group calls even where OpenTelemetry is not configured.

```typescript
import { context, propagation, trace, SpanStatusCode } from "@opentelemetry/api";

const tracer = trace.getTracer("support-agent");
const GATEWAY = "https://api.infro.io/v1/chat/completions";

type Step = "plan" | "tool_select" | "answer" | "critique"; // a fixed, small vocabulary

async function modelCall(runId: string, step: Step, body: Record<string, unknown>) {
  return tracer.startActiveSpan(`chat ${String(body.model)}`, async (span) => {
    span.setAttribute("gen_ai.operation.name", "chat");
    span.setAttribute("gen_ai.request.model", String(body.model));
    span.setAttribute("app.step", step);

    const headers: Record<string, string> = {
      Authorization: `Bearer ${process.env.INFRO_API_KEY}`,
      "Content-Type": "application/json",
    };
    propagation.inject(context.active(), headers); // adds traceparent (and tracestate)

    try {
      const res = await fetch(GATEWAY, {
        method: "POST",
        headers,
        body: JSON.stringify({
          ...body,
          metadata: { trace_id: runId, workflow: "support-agent", step },
        }),
      });
      const requestId = res.headers.get("X-INFRO-Request-Id");
      if (requestId) span.setAttribute("infro.request_id", requestId);
      if (!res.ok) throw new Error(`gateway ${res.status}: ${await res.text()}`);

      const json = await res.json();
      span.setAttribute("gen_ai.usage.input_tokens", json.usage?.prompt_tokens ?? 0);
      span.setAttribute("gen_ai.usage.output_tokens", json.usage?.completion_tokens ?? 0);
      return json;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

async function toolCall<T>(name: string, callId: string, fn: () => Promise<T>): Promise<T> {
  return tracer.startActiveSpan(`execute_tool ${name}`, async (span) => {
    span.setAttribute("gen_ai.operation.name", "execute_tool");
    span.setAttribute("gen_ai.tool.name", name);
    span.setAttribute("gen_ai.tool.call.id", callId);
    try {
      return await fn();
    } finally {
      span.end();
    }
  });
}

export async function runAgent(task: string) {
  return tracer.startActiveSpan("invoke_agent support-agent", async (root) => {
    // The OTel trace id doubles as the run id, so both systems agree.
    const runId = root.spanContext().traceId;
    root.setAttribute("gen_ai.agent.name", "support-agent");
    try {
      const plan = await modelCall(runId, "plan", {
        model: "anthropic/claude-sonnet-5",
        messages: [{ role: "user", content: task }],
        max_tokens: 800,
      });
      // ... toolCall(...) and further modelCall(...) steps ...
      return { runId, plan };
    } finally {
      root.end();
    }
  });
}
```

Two decisions in that code are worth copying. The `step` vocabulary is a closed type: free-text step names drift (`answer`, `final_answer`, `Answer`) and split one step across three groups. And the gateway's request id is recorded on the span, so an engineer looking at a slow span can open the exact gateway record behind it.

[Diagram: The same identity travels in the traceparent header and in request metadata. The gateway's spans can land in your collector nested under your own model-call span.]

## Reading a run in INFRO

INFRO records every model call routed through it with the model, the provider account that served it, fallback attempts, timings, token usage and an estimate of what your provider will bill for it. Four reserved `metadata` keys turn those records into runs ([request traces](https://infro.io/docs/observability/traces)):

| Key | Meaning | Shape |
| --- | --- | --- |
| `trace_id` | One run | Up to 128 letters, digits and `_ . : -` |
| `workflow` | The stable name of the work, such as `support-agent` | Up to 64 characters, same set |
| `step` | This call's role in the run, such as `plan` or `answer` | Up to 64 characters, same set |
| `feature` | The product feature the call serves | Up to 64 characters, same set |

Table: Reserved metadata keys, from the request traces documentation.

A malformed reserved key is refused with `400 invalid_attribution` naming the key, rather than silently dropped — a refusal on the first request of an integration is easier to fix than a hole in a cost report weeks later. Requests that cannot carry a JSON body can send `X-INFRO-Trace-Id`, `X-INFRO-Workflow`, `X-INFRO-Step` and `X-INFRO-Feature` headers instead.

[Diagram: Body metadata wins, then headers, then a valid traceparent. A traceparent also nests INFRO's spans in your trace whichever source supplied the trace id.]

Precedence is body metadata, then headers, then `traceparent`. In the gateway source, a malformed `traceparent` is treated as absent rather than refused, which matches the W3C rule above. One consequence is easy to miss: a `metadata.trace_id` sent *without* a `traceparent` groups requests in INFRO but does not become an OpenTelemetry trace id, because inventing trace context from an arbitrary string could join unrelated traces in your backend. Send `traceparent` if you want INFRO's spans inside your own trace.

`GET /v1/traces/{trace_id}` returns every request in one run, in order, with totals:

```json
{
  "trace_id": "run_8f2c",
  "workflows": ["support-agent"],
  "started_at": "2026-09-29T09:14:02.118Z",
  "ended_at": "2026-09-29T09:14:31.904Z",
  "requests": [{ "id": "req_…", "step": "plan", "model": "anthropic/claude-sonnet-5", "…": "…" }],
  "truncated": false,
  "totals": {
    "requests": 9,
    "failed": 1,
    "estimated_cost": 0.0841,
    "unpriced_requests": 0,
    "cost_is_partial": false,
    "by_step": [{ "step": "answer", "requests": 3, "estimated_cost": 0.051, "share": 0.61 }],
    "by_modality": [{ "modality": "text", "requests": 9, "estimated_cost": 0.0841, "share": 1 }],
    "providers": ["anthropic"]
  },
  "covers": "model_calls_through_infro"
}
```

Read the last field first. The total covers model calls made through INFRO and nothing else: your vector database, your retrieval and your own compute are not in it ([request traces](https://infro.io/docs/observability/traces)). `workflows` is a list because trace ids are yours to generate and nothing stops two workflows sharing one by accident. When any successful call in the run had no price, `cost_is_partial` is true and the figure is a lower bound, never a filled-in guess. In the current gateway source, one answer returns up to 500 requests of a run and sets `truncated` beyond that. The same records are searchable in the console with `trace:<id>`, and `GET /v1/requests/export?trace_id=…` downloads one run as CSV without prompts or responses.

For cost across many runs, `GET /v1/usage?group_by=step` (or `workflow`, `feature`) aggregates the same estimates; a row per run is what the trace endpoint is for.

## Sending spans to your own backend

On the Team plan and above, INFRO can deliver OTLP/HTTP traces to your collector's `/v1/traces` endpoint, with service name `infro-gateway` ([exports](https://infro.io/docs/observability/exports)). Spans carry operational metadata — request id, model, route, timing, token counts, estimated cost — plus the attribution values as `infro.trace_id`, `infro.workflow`, `infro.step` and `infro.feature`, and never captured prompts or responses. In the gateway source they also carry `gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`, and `gen_ai.provider.name` only for provider kinds whose conventional value is known.

Delivery is best effort and outside the response path: one retry on a server or network error, none on a `4xx`, and dropped spans rather than blocked requests when the collector keeps failing. Treat the exported spans as observability, not as the accounting record; the request API and CSV export are the record.

## Common tracing mistakes

- **A new trace id per model call.** The run disappears into unrelated fragments. Generate the id once, at the root.
- **Reusing a request id as the run id.** Frameworks that recycle request ids merge unrelated runs. Use a random, opaque id.
- **Step names with parameters in them.** `search_customer_123` creates a group per customer. Put parameters in span attributes, not in the step name.
- **Personal data in metadata.** Trace ids and step names end up in URLs, exports and dashboards. Keep them opaque; send user identifiers as opaque ids.
- **Presenting a gateway total as the run's cost.** It is the model-call estimate. Retrieval, tools and compute are separate lines.

## Checklist

- [ ] Each run has one trace id, created at the root and never regenerated by retries or sub-agents.
- [ ] Every outbound model call carries `traceparent` and the run id in metadata.
- [ ] Step names come from a fixed vocabulary per workflow.
- [ ] Tool calls, retrieval and queue waits have their own spans.
- [ ] Each model-call span records the gateway request id.
- [ ] Dashboards show cost and latency by step, and label run totals with what they cover.
- [ ] The GenAI attribute names your alerts depend on are pinned and re-checked on upgrade.

## Questions

### Should I use my own run id or the OpenTelemetry trace id?

If you already run OpenTelemetry, let its trace id be the run id and propagate it with traceparent, so model calls, tools and databases join one trace. If you do not, generate an opaque run id yourself and send it as metadata; you can adopt traceparent later without changing how runs are grouped.

## Sources

1. Building effective agents — Anthropic. https://www.anthropic.com/engineering/building-effective-agents (accessed 2026-09-29)
2. Trace Context (W3C Recommendation) — W3C. https://www.w3.org/TR/trace-context/ (accessed 2026-09-29)
3. Semantic conventions for generative AI systems (moved notice) — OpenTelemetry. https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/ (accessed 2026-09-29)
4. Semantic conventions for generative AI client spans — OpenTelemetry. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md (accessed 2026-09-29)
5. Semantic conventions for GenAI agent and framework spans — OpenTelemetry. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md (accessed 2026-09-29)