# Measuring time to first token and setting timeouts

URL: https://infro.io/blog/measure-time-to-first-token
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-09-29
Last modified: 2026-09-29
Last reviewed: 2026-09-29 (next review due 2026-10-29)
Topic: Routing and reliability
Format: Methodology

> Measure time to first token from the moment the client sends the request to the first delta carrying content or a tool call, not to the first byte or first event. Record it per request with the model, route and prompt size, read it as percentiles over successful requests, and set a first-token timeout from the tail with headroom, inside a total deadline that leaves room for failover.

For a streamed answer, the wait that users notice is the silence before the first words appear. After that the text is arriving and the experience is reading, not waiting. That makes time to first token (TTFT) the latency number that matters most for interactive features — and also the one most often measured at the wrong point, averaged, and then turned into a timeout by guesswork.

This is a method for doing it properly: one definition, instrumentation at two points, a way to read the data, and a rule for deriving timeouts from it.

## Define the two ends of the clock

**Start** when the client issues the request, before connection setup. Starting after the connection is open hides DNS, TLS and connection-pool waits, which are real parts of what the user waits for.

**Stop** at the first chunk that carries something usable: a content delta or a tool-call delta. Not the response headers, and not the first event. Many streams open with events that carry no output: an OpenAI-compatible stream's first chunk typically sets only the role, and an Anthropic stream begins with `message_start` and may include any number of `ping` events[1]. A model that reasons before answering can send those promptly and then think for a long time.

OpenTelemetry's GenAI conventions define a client metric, `gen_ai.client.operation.time_to_first_chunk`, measured from when the client issues the request to when the first chunk arrives, and a separate `gen_ai.client.operation.time_per_output_chunk` for the gaps after it[2]. Both are marked as in development. If you adopt them, note that a *chunk* is not a token: record the first-content time as well, or your dashboards will reward a model for sending a role-only chunk quickly.

[Diagram: Everything between sending the request and the first usable delta is time to first token. The first event is often not the first token, and every failed attempt before it adds its full duration.]

## Record the context with every measurement

A TTFT number without context cannot be compared. Record, per request:

- **Target:** the model, the route or configuration name, and the provider account that served.
- **Prompt size:** input tokens, and the cached share where the provider reports it. Longer prompts take longer to process before the first token, so mixing prompt sizes blurs every comparison.
- **Outcome:** success, error, interrupted or timed out. Google's SRE book recommends keeping the latency of failed requests separate from successful ones, because a fast error flatters the average and a slow error is worse than a fast one[3].
- **Attempts:** whether the request failed over before it started streaming. A recovered request can succeed and still blow the latency budget.
- **The rest of the timeline:** headers time, first event time, total time and output tokens, so you can tell a slow start from a slow generation.

## Measure on the client

The function below times one OpenAI-compatible streaming request and enforces three limits at once: a first-token timeout, an idle timeout between chunks, and a total deadline. It stops the TTFT clock only on a content or tool-call delta.

```typescript
export type Outcome =
  | "complete" | "error" | "closed"
  | "timeout_first_token" | "timeout_idle" | "timeout_deadline";

export interface StreamTiming {
  requestId: string | null;
  headersMs: number | null;
  firstEventMs: number | null;
  ttftMs: number | null;     // first content or tool-call delta
  totalMs: number;
  contentChunks: number;
  outcome: Outcome;
}

export async function timedStream(
  url: string,
  apiKey: string,
  body: Record<string, unknown>,
  limits = { firstTokenMs: 10_000, idleMs: 60_000, deadlineMs: 120_000 },
): Promise<StreamTiming> {
  const t0 = performance.now();
  const since = () => Math.round(performance.now() - t0);
  const controller = new AbortController();
  let reason: Outcome | null = null;
  const abort = (why: Outcome) => {
    reason ??= why;
    controller.abort();
  };

  const deadline = setTimeout(() => abort("timeout_deadline"), limits.deadlineMs);
  const firstToken = setTimeout(() => abort("timeout_first_token"), limits.firstTokenMs);
  let idle: ReturnType<typeof setTimeout> | undefined;

  const timing: StreamTiming = {
    requestId: null, headersMs: null, firstEventMs: null, ttftMs: null,
    totalMs: 0, contentChunks: 0, outcome: "closed",
  };
  let sawDone = false;

  try {
    const res = await fetch(url, {
      method: "POST",
      headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
      body: JSON.stringify({ ...body, stream: true }),
      signal: controller.signal,
    });
    timing.headersMs = since();
    timing.requestId = res.headers.get("x-infro-request-id") ?? res.headers.get("x-request-id");
    if (!res.ok || !res.body) {
      timing.outcome = "error";
      return timing;
    }

    const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
    let buffer = "";
    for (;;) {
      const { value, done } = await reader.read();
      if (done) break;
      clearTimeout(idle);
      idle = setTimeout(() => abort("timeout_idle"), limits.idleMs);
      buffer += value;

      let nl: number;
      while ((nl = buffer.indexOf("\n")) !== -1) {
        const line = buffer.slice(0, nl).trimEnd();
        buffer = buffer.slice(nl + 1);
        if (!line.startsWith("data:")) continue;       // comments, blank lines
        const data = line.slice(5).trim();
        if (data === "[DONE]") { sawDone = true; continue; }

        timing.firstEventMs ??= since();
        const event = JSON.parse(data);
        if (event.error) { timing.outcome = "error"; return timing; }

        const delta = event.choices?.[0]?.delta;
        if (delta?.content || delta?.tool_calls) {
          if (timing.ttftMs === null) {
            timing.ttftMs = since();
            clearTimeout(firstToken);
          }
          timing.contentChunks += 1;
        }
      }
    }
    timing.outcome = sawDone ? "complete" : "closed";
  } catch (err) {
    if (reason) timing.outcome = reason;
    else if (timing.headersMs === null) throw err;     // never connected: a transport error
    else timing.outcome = "closed";
  } finally {
    clearTimeout(deadline);
    clearTimeout(firstToken);
    clearTimeout(idle);
    controller.abort();                                 // release the connection
    timing.totalMs = since();
  }
  return timing;
}
```

Emit every field of the returned object, including the outcome, to your metrics pipeline as a histogram per target. Use a monotonic clock, as `performance.now()` is here; wall-clock time can step backwards under clock adjustment.

## Read it as percentiles, sliced

Averages hide the tail users remember. Google's SRE book gives the example of a service averaging 100 ms at 1,000 requests a second where 1% of requests could easily take five seconds, and recommends histograms with exponentially spaced buckets instead of means[3]. For TTFT, read at least p50, p90, p99 and, with enough volume, p99.9 — per target and per prompt-size band, over successful requests only.

The script below does that from a request export. The column names are those of INFRO's CSV export ([traces](https://infro.io/docs/observability/traces)); any log with the same fields works.

```python
import csv
import math
from collections import defaultdict


def pct(sorted_values: list[float], p: float) -> float:
    """Nearest-rank percentile of an already sorted list."""
    k = max(0, math.ceil(p / 100 * len(sorted_values)) - 1)
    return sorted_values[k]


def prompt_band(tokens: str) -> str:
    n = int(float(tokens or 0))
    if n < 2_000:
        return "<2k"
    if n < 16_000:
        return "2k-16k"
    if n < 64_000:
        return "16k-64k"
    return ">=64k"


groups: dict[tuple[str, str], list[float]] = defaultdict(list)
with open("requests.csv", newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        if row["status"] != "ok" or not row["ttft_ms"]:
            continue                      # successful, streamed requests only
        target = row["route_slug"] or row["model"]
        groups[(target, prompt_band(row["prompt_tokens"]))].append(float(row["ttft_ms"]))

for (target, band), values in sorted(groups.items()):
    values.sort()
    if len(values) < 200:
        print(f"{target:30} {band:8} n={len(values):>6}  too few for a tail estimate")
        continue
    p50, p99, p999 = pct(values, 50), pct(values, 99), pct(values, 99.9)
    print(f"{target:30} {band:8} n={len(values):>6}  "
          f"p50={p50:>7.0f}  p99={p99:>7.0f}  p99.9={p999:>7.0f} ms")
```

Two reading rules. A p99.9 from a few hundred requests is one or two data points; wait for volume before you set a limit from it. And compare like with like: a model that looks slow may simply be serving the route with the longest prompts.

## Turn the tail into timeouts

A streamed request needs three limits, and each answers a different question.

[Diagram: Three limits for one streamed request. Only the total deadline comes from the caller; the other two come from measurement.]

**The first-token timeout** decides when a silent request is abandoned. Set it from the tail of successful TTFT for that target and prompt band, with headroom — a limit at p99.9 fails roughly one healthy request in a thousand, which is the rate you are choosing to fail over. A timeout at the median fails half your traffic, and one at ten times p99.9 never fires before the user has left.

[Diagram: Illustrative figures for one route: a first-token timeout chosen a little above p99.9 of successful requests, not from the mean.]

**The idle timeout** decides when a started stream is dead. It must be longer than the longest legitimate pause between chunks, which for reasoning models can be long.

**The total deadline** is how long the caller is still waiting. Every attempt has to fit inside it, and a request that has already missed its deadline should not start another attempt; Google's SRE guidance on overload makes the related point that retries should be bounded so they cannot multiply load during an incident[4].

> **Example: Fitting a first-token timeout inside a deadline**
> A chat feature gives up after 30 seconds. Its route's p99.9 TTFT is 7.5 seconds, so the first-token timeout is set to 10 seconds. If the first target stalls, it costs 10 seconds and the second target starts with 20 seconds left, which covers its first token and a typical answer. If the timeout were 25 seconds, a stalled first target would leave the second only 5 seconds, and failover would rarely finish in time. These figures are illustrative; take yours from your own percentiles.

Revisit the limits when anything that moves TTFT changes: a new model or snapshot, a longer system prompt, a provider incident, or a new region.

## Where INFRO starts and stops its clock

INFRO records its own time to first token on streamed chat completions, so you can compare the client's view with the gateway's. It is returned as `first_token_ms` on `GET /v1/requests/{id}`, beside `latency_ms`, and exported as `ttft_ms` in the request CSV ([traces](https://infro.io/docs/observability/traces)).

- **Start:** when the gateway begins handling the request. The difference between your client TTFT and INFRO's is network time between your application and the gateway.
- **Stop:** when the gateway forwards the first delta carrying content or a tool call — the same rule as the client code above. With an output [guardrail](https://infro.io/docs/features/guardrails), text is held back briefly before release, so the clock stops at release.
- **Failover is inside it.** Attempts that failed before the stream opened are part of the wait, and the request record lists each with its error and latency; the answering connection is named on every chunk ([streaming](https://infro.io/docs/api/streaming)).
- **Non-streamed requests** have no first-token time; compare models on streamed traffic.

On timeouts, the gateway's limits are deliberately generous and yours should be the tighter ones. A [route](https://infro.io/docs/routes) can set a per-attempt timeout between 1 second and 10 minutes, which, for a stream, bounds the wait for the provider's response; after that, silence is bounded by an idle limit that is never less than 120 seconds, both before the first event, where the request still fails over, and after it, where the stream ends with `stream_stalled`. If your client's first-token timeout is shorter than the route's per-attempt timeout, the client gives up before the gateway can fail over, so size the two together.

For trends rather than single requests, [traffic changes](https://infro.io/docs/observability/traffic-changes) watch p95 latency and p95 time to first token per scope, and raise a finding only when a slowdown persists for two days rather than for one bad afternoon.

## Checklist

- [ ] Start the clock before connection setup; stop it at the first content or tool-call delta.
- [ ] Record target, prompt size, outcome and attempts with every measurement.
- [ ] Keep successful and failed requests in separate histograms.
- [ ] Read p50, p99 and p99.9 per target and prompt band, never the mean alone.
- [ ] Set first-token and idle timeouts per target from measured tails, with headroom.
- [ ] Check that two attempts fit inside the caller's deadline.
- [ ] Size the client's first-token timeout together with the gateway's per-attempt timeout.
- [ ] Re-derive limits after a model, prompt or provider change.

## Questions

### Is time to first token the same as time to first byte?

No. The first byte is usually response headers, and the first streamed event may carry only a role or a keep-alive. Time to first token should stop at the first chunk that carries content or a tool call, which can be much later, especially for models that reason before answering.

### Should a non-streaming request have a time to first token?

Not a meaningful one. Without streaming the client receives nothing until the whole answer is ready, so measure total latency and time to response headers instead, and compare models on streamed traffic.

## Sources

1. Streaming messages — Anthropic. https://platform.claude.com/docs/en/build-with-claude/streaming (accessed 2026-09-29)
2. Semantic conventions for generative AI metrics — OpenTelemetry. https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md (accessed 2026-09-29)
3. Monitoring Distributed Systems — Site Reliability Engineering — Google. https://sre.google/sre-book/monitoring-distributed-systems/ (accessed 2026-09-29)
4. Handling Overload — Site Reliability Engineering — Google. https://sre.google/sre-book/handling-overload/ (accessed 2026-09-29)