Measuring time to first token and setting timeouts

For a streamed answer, the wait that users notice is the silence before the first words appear. After that the text is arriving and the experience is reading, not waiting. That makes time to first token (TTFT) the latency number that matters most for interactive features — and also the one most often measured at the wrong point, averaged, and then turned into a timeout by guesswork.

This is a method for doing it properly: one definition, instrumentation at two points, a way to read the data, and a rule for deriving timeouts from it.

Define the two ends of the clock

Start when the client issues the request, before connection setup. Starting after the connection is open hides DNS, TLS and connection-pool waits, which are real parts of what the user waits for.

Stop at the first chunk that carries something usable: a content delta or a tool-call delta. Not the response headers, and not the first event. Many streams open with events that carry no output: an OpenAI-compatible stream's first chunk typically sets only the role, and an Anthropic stream begins with message_start and may include any number of ping events¹. A model that reasons before answering can send those promptly and then think for a long time.

OpenTelemetry's GenAI conventions define a client metric, gen_ai.client.operation.time_to_first_chunk, measured from when the client issues the request to when the first chunk arrives, and a separate gen_ai.client.operation.time_per_output_chunk for the gaps after it². Both are marked as in development. If you adopt them, note that a chunk is not a token: record the first-content time as well, or your dashboards will reward a model for sending a role-only chunk quickly.

Record the context with every measurement

A TTFT number without context cannot be compared. Record, per request:

• Target: the model, the route or configuration name, and the provider account that served.
• Prompt size: input tokens, and the cached share where the provider reports it. Longer prompts take longer to process before the first token, so mixing prompt sizes blurs every comparison.
• Outcome: success, error, interrupted or timed out. Google's SRE book recommends keeping the latency of failed requests separate from successful ones, because a fast error flatters the average and a slow error is worse than a fast one³.
• Attempts: whether the request failed over before it started streaming. A recovered request can succeed and still blow the latency budget.
• The rest of the timeline: headers time, first event time, total time and output tokens, so you can tell a slow start from a slow generation.

Measure on the client

The function below times one OpenAI-compatible streaming request and enforces three limits at once: a first-token timeout, an idle timeout between chunks, and a total deadline. It stops the TTFT clock only on a content or tool-call delta.

export type Outcome =
  | "complete" | "error" | "closed"
  | "timeout_first_token" | "timeout_idle" | "timeout_deadline";

export interface StreamTiming {
  requestId: string | null;
  headersMs: number | null;
  firstEventMs: number | null;
  ttftMs: number | null;     // first content or tool-call delta
  totalMs: number;
  contentChunks: number;
  outcome: Outcome;
}

export async function timedStream(
  url: string,
  apiKey: string,
  body: Record<string, unknown>,
  limits = { firstTokenMs: 10_000, idleMs: 60_000, deadlineMs: 120_000 },
): Promise<StreamTiming> {
  const t0 = performance.now();
  const since = () => Math.round(performance.now() - t0);
  const controller = new AbortController();
  let reason: Outcome | null = null;
  const abort = (why: Outcome) => {
    reason ??= why;
    controller.abort();
  };

  const deadline = setTimeout(() => abort("timeout_deadline"), limits.deadlineMs);
  const firstToken = setTimeout(() => abort("timeout_first_token"), limits.firstTokenMs);
  let idle: ReturnType<typeof setTimeout> | undefined;

  const timing: StreamTiming = {
    requestId: null, headersMs: null, firstEventMs: null, ttftMs: null,
    totalMs: 0, contentChunks: 0, outcome: "closed",
  };
  let sawDone = false;

  try {
    const res = await fetch(url, {
      method: "POST",
      headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
      body: JSON.stringify({ ...body, stream: true }),
      signal: controller.signal,
    });
    timing.headersMs = since();
    timing.requestId = res.headers.get("x-infro-request-id") ?? res.headers.get("x-request-id");
    if (!res.ok || !res.body) {
      timing.outcome = "error";
      return timing;
    }

    const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
    let buffer = "";
    for (;;) {
      const { value, done } = await reader.read();
      if (done) break;
      clearTimeout(idle);
      idle = setTimeout(() => abort("timeout_idle"), limits.idleMs);
      buffer += value;

      let nl: number;
      while ((nl = buffer.indexOf("\n")) !== -1) {
        const line = buffer.slice(0, nl).trimEnd();
        buffer = buffer.slice(nl + 1);
        if (!line.startsWith("data:")) continue;       // comments, blank lines
        const data = line.slice(5).trim();
        if (data === "[DONE]") { sawDone = true; continue; }

        timing.firstEventMs ??= since();
        const event = JSON.parse(data);
        if (event.error) { timing.outcome = "error"; return timing; }

        const delta = event.choices?.[0]?.delta;
        if (delta?.content || delta?.tool_calls) {
          if (timing.ttftMs === null) {
            timing.ttftMs = since();
            clearTimeout(firstToken);
          }
          timing.contentChunks += 1;
        }
      }
    }
    timing.outcome = sawDone ? "complete" : "closed";
  } catch (err) {
    if (reason) timing.outcome = reason;
    else if (timing.headersMs === null) throw err;     // never connected: a transport error
    else timing.outcome = "closed";
  } finally {
    clearTimeout(deadline);
    clearTimeout(firstToken);
    clearTimeout(idle);
    controller.abort();                                 // release the connection
    timing.totalMs = since();
  }
  return timing;
}

Emit every field of the returned object, including the outcome, to your metrics pipeline as a histogram per target. Use a monotonic clock, as performance.now() is here; wall-clock time can step backwards under clock adjustment.

Read it as percentiles, sliced

Averages hide the tail users remember. Google's SRE book gives the example of a service averaging 100 ms at 1,000 requests a second where 1% of requests could easily take five seconds, and recommends histograms with exponentially spaced buckets instead of means³. For TTFT, read at least p50, p90, p99 and, with enough volume, p99.9 — per target and per prompt-size band, over successful requests only.

The script below does that from a request export. The column names are those of INFRO's CSV export (traces); any log with the same fields works.

import csv
import math
from collections import defaultdict


def pct(sorted_values: list[float], p: float) -> float:
    """Nearest-rank percentile of an already sorted list."""
    k = max(0, math.ceil(p / 100 * len(sorted_values)) - 1)
    return sorted_values[k]


def prompt_band(tokens: str) -> str:
    n = int(float(tokens or 0))
    if n < 2_000:
        return "<2k"
    if n < 16_000:
        return "2k-16k"
    if n < 64_000:
        return "16k-64k"
    return ">=64k"


groups: dict[tuple[str, str], list[float]] = defaultdict(list)
with open("requests.csv", newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        if row["status"] != "ok" or not row["ttft_ms"]:
            continue                      # successful, streamed requests only
        target = row["route_slug"] or row["model"]
        groups[(target, prompt_band(row["prompt_tokens"]))].append(float(row["ttft_ms"]))

for (target, band), values in sorted(groups.items()):
    values.sort()
    if len(values) < 200:
        print(f"{target:30} {band:8} n={len(values):>6}  too few for a tail estimate")
        continue
    p50, p99, p999 = pct(values, 50), pct(values, 99), pct(values, 99.9)
    print(f"{target:30} {band:8} n={len(values):>6}  "
          f"p50={p50:>7.0f}  p99={p99:>7.0f}  p99.9={p999:>7.0f} ms")

Two reading rules. A p99.9 from a few hundred requests is one or two data points; wait for volume before you set a limit from it. And compare like with like: a model that looks slow may simply be serving the route with the longest prompts.

Turn the tail into timeouts

A streamed request needs three limits, and each answers a different question.

The first-token timeout decides when a silent request is abandoned. Set it from the tail of successful TTFT for that target and prompt band, with headroom — a limit at p99.9 fails roughly one healthy request in a thousand, which is the rate you are choosing to fail over. A timeout at the median fails half your traffic, and one at ten times p99.9 never fires before the user has left.

The idle timeout decides when a started stream is dead. It must be longer than the longest legitimate pause between chunks, which for reasoning models can be long.

The total deadline is how long the caller is still waiting. Every attempt has to fit inside it, and a request that has already missed its deadline should not start another attempt; Google's SRE guidance on overload makes the related point that retries should be bounded so they cannot multiply load during an incident⁴.

Revisit the limits when anything that moves TTFT changes: a new model or snapshot, a longer system prompt, a provider incident, or a new region.

Where INFRO starts and stops its clock

INFRO records its own time to first token on streamed chat completions, so you can compare the client's view with the gateway's. It is returned as first_token_ms on GET /v1/requests/{id}, beside latency_ms, and exported as ttft_ms in the request CSV (traces).

• Start: when the gateway begins handling the request. The difference between your client TTFT and INFRO's is network time between your application and the gateway.
• Stop: when the gateway forwards the first delta carrying content or a tool call — the same rule as the client code above. With an output guardrail, text is held back briefly before release, so the clock stops at release.
• Failover is inside it. Attempts that failed before the stream opened are part of the wait, and the request record lists each with its error and latency; the answering connection is named on every chunk (streaming).
• Non-streamed requests have no first-token time; compare models on streamed traffic.

On timeouts, the gateway's limits are deliberately generous and yours should be the tighter ones. A route can set a per-attempt timeout between 1 second and 10 minutes, which, for a stream, bounds the wait for the provider's response; after that, silence is bounded by an idle limit that is never less than 120 seconds, both before the first event, where the request still fails over, and after it, where the stream ends with stream_stalled. If your client's first-token timeout is shorter than the route's per-attempt timeout, the client gives up before the gateway can fail over, so size the two together.

For trends rather than single requests, traffic changes watch p95 latency and p95 time to first token per scope, and raise a finding only when a slowdown persists for two days rather than for one bad afternoon.

Checklist

• Start the clock before connection setup; stop it at the first content or tool-call delta.
• Record target, prompt size, outcome and attempts with every measurement.
• Keep successful and failed requests in separate histograms.
• Read p50, p99 and p99.9 per target and prompt band, never the mean alone.
• Set first-token and idle timeouts per target from measured tails, with headroom.
• Check that two attempts fit inside the caller's deadline.
• Size the client's first-token timeout together with the gateway's per-attempt timeout.
• Re-derive limits after a model, prompt or provider change.