LLM observability: what to measure in production

Teams often discover the limits of their AI monitoring the same way. A dashboard says error rates are flat and latency is normal, while support tickets say the assistant is producing nonsense, a batch job's bill has doubled, or answers take twenty seconds to start. Each of those is a failure an ordinary API dashboard cannot see: the status code was 200, the average latency was fine, and cost was not a metric at all.

This guide describes what to record for every LLM request, the signals to build on those records, how to connect them to the rest of your tracing, and how to do it without storing people's prompts.

Why API monitoring misses LLM failures

Four properties of LLM traffic break the assumptions a generic HTTP dashboard makes:

• Success is not correctness. A response can arrive with status 200 and still fail its purpose: invalid JSON, a hallucinated tool argument, a refusal, an answer from a fallback model that was never evaluated for the task.
• Latency has two shapes. For streamed responses, the time until the first token decides whether a product feels responsive; the total duration decides throughput. One number cannot represent both.
• Cost varies per request. Two requests to the same endpoint can differ in cost by orders of magnitude, driven by prompt length, output length, cache hits and which account served them.
• One request can be several attempts, and several calls. When a gateway fails over, the successful response hides the failed attempts that preceded it — and their latency and, sometimes, their cost. A request can also trigger calls of its own: a classifier that picks the route, a safety check on the prompt, a background evaluation of the answer. Each costs money and can fail, and none of them is the request the user made.

Adapt the four golden signals

Google's SRE guidance names four golden signals for monitoring a user-facing system: latency, traffic, errors and saturation¹. Each needs an LLM-specific definition.

The request record is the unit of observability

A log line per request is easy to produce and hard to use. What operators need is one structured record per request that answers "what happened, why, and what did it cost" without joining three systems.

Two fields deserve emphasis. Model requested versus model served is how you detect a silent substitution — an alias that resolved differently, or a fallback that answered with another model. Record a third value too, the model id the provider's response names, because providers move a model name to a new snapshot without changing the name you send. The request id must be returned to the caller, so a support ticket, an application log and the gateway record can all be joined on one value.

Calls a request triggers belong in the same store, as rows of their own linked to the request they served, with their own cost and outcome. Folded into the parent, a classifier's timeouts and its share of spend disappear; the argument is made at length in classifiers in the request path.

Connect LLM records to the rest of your traces

An LLM call is usually one span inside a larger request. W3C Trace Context standardizes the traceparent and tracestate headers that carry trace identity between services², so a gateway that accepts and propagates them lets a model call appear inside the trace of the user action that caused it.

OpenTelemetry is developing semantic conventions for generative AI that name the attributes such spans should carry — for example gen_ai.request.model, gen_ai.response.model, gen_ai.provider.name, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens³. The conventions are marked as in development, so expect names to change and pin the version your collector expects.

Treat cost as a telemetry signal

Model responses report usage: the Chat Completions response includes prompt and completion token counts, with cached prompt tokens broken out when prompt caching applied⁴. Anthropic reports cache writes and cache reads as separate fields, and prices them differently from ordinary input tokens⁵. Turning those counts into cost at the time of the request — rather than at the end of the month — makes cost a signal you can alert on and attribute.

Three rules keep it trustworthy:

1. Use the rate your account pays. A negotiated rate, or a cloud platform's rate card, can differ from the model vendor's list price. Record which rate was used.
2. Record unknown as unknown. A model with no rate on file should produce an empty cost, not zero. Zero reads as free and quietly understates spend.
3. Keep the estimate separate from the invoice. A per-request estimate is what budgets and attribution are computed from; the provider's invoice is the authority. Compare them periodically and investigate a sustained gap.

If anything in front of the provider answers requests itself — an exact-match response cache, for instance — mark those responses. A cache hit costs the provider nothing and returns in milliseconds; mixed into the same series, it flatters both p95 latency and cost per request, and hides a provider getting slower. Report hits against lookups, and their latency apart from provider latency. When to use such a cache at all is covered in exact-match response caching.

Quality signals that do not require reading prompts

Most quality regressions can be detected from signals the application computes anyway:

• the share of responses that fail schema validation or parsing;
• tool calls with invalid arguments, and tools that returned errors;
• refusals and truncated outputs (a finish reason of length rather than stop);
• retries triggered by the application itself;
• user corrections, regenerations and escalations to a person;
• scores from a sampled, access-controlled evaluation run.

Record these as metadata on the request or as events joined by request id. They catch the release that made a prompt worse without anyone reading a customer's conversation.

Changes that arrive with status 200

Some regressions are not caused by your release at all. A provider moves a model name to a new snapshot and answers get longer. A small edit near the top of a prompt stops the provider's prompt cache from matching, and input cost rises. A primary starts timing out for a few requests, and they quietly move to a fallback with a higher rate. None of these produces an error, and each is visible in metadata alone:

Compare each against its own recent baseline, per model and per account, and require both a relative and an absolute change before alerting, or every quiet night becomes a finding. What metadata cannot tell you is whether answers got worse; that needs an evaluation, as described in how to evaluate LLMs on real workloads.

Metrics that should change a decision

A metric earns a place on a dashboard when crossing a threshold triggers an action someone owns.

Privacy, retention and access

Request metadata can reveal a great deal — which customers use a feature, when, and how heavily — and prompts and completions can contain exactly the personal and confidential data that sensitive-information-disclosure risks describe⁶. Decide the policy before the first incident, when there is time to think:

• Keep prompt and completion capture off by default. Enable it for a defined purpose, a defined scope and a defined retention window.
• Keep operational metadata longer than content. Most investigations need the path, timing and error, not the text.
• Redact secrets before anything is logged, including inside tool arguments, and make sure the redacted text is what every downstream copy sees: retries, captures and exported spans.
• Apply the same role boundaries to reading request data as to managing production keys.

An annotated request record

The record below is illustrative and abbreviated, in the shape INFRO returns. Each field answers one of the questions above.

{
  "id": "req_…",
  "status": "ok",
  "http_status": 200,
  "model_requested": "support-drafts",
  "model": "anthropic/claude-sonnet-5",
  "served_model": "claude-sonnet-5",
  "finish_reason": "stop",
  "route": "primary",
  "route_slug": "support-drafts",
  "provider": "anthropic",
  "connection_name": "Anthropic production",
  "fallback_attempts": [],
  "latency_ms": 4210,
  "first_token_ms": 640,
  "usage": {
    "prompt_tokens": 1830,
    "completion_tokens": 412,
    "total_tokens": 2242,
    "cost": 0.01167,
    "price_source": "list"
  },
  "metadata": { "user_id": "cust_8231", "tags": { "feature": "ticket-reply" } }
}

• model_requested is a route slug and model is what served it, so a substitution would be visible here.
• served_model is the id the provider's response named; a new value here, with the same model, is a snapshot change.
• finish_reason is stop, not length, so the answer was not cut off.
• fallback_attempts is empty: nothing failed before this response.
• first_token_ms and latency_ms separate a slow start from a long answer.
• price_source says the estimate used the vendor's list rate, not a negotiated one.
• metadata carries attribution you supplied, without any prompt content.

How INFRO records requests

Every request through INFRO produces a record like the one above, including refused and failed requests once the gateway knows which model was asked for. Every response carries an X-INFRO-Request-Id header with the same id. The records are about traffic on your own provider accounts; every cost in them is an estimate of what your provider will bill, null when the model is unpriced, and never includes INFRO's subscription.

• GET /v1/requests lists records with filters for status, model, project, key, provider, connection, route, and the user_id, session_id and tag.<key> values you send in request metadata; sort can be newest, slowest or costliest. GET /v1/requests/{id} returns one record with its timing spans.
• GET /v1/usage aggregates requests, units, estimated cost, unpriced requests, error rate and p50, p95 and p99 latency by hour, day or month, grouped by model, modality, project, key, endpoint, provider, connection or route. Latency is measured on calls that reached your provider.
• Calls INFRO makes for a request — a guardrail classifier check, a route's prompt classifier, an eval monitor's score — are rows of their own with their own estimated cost, carrying purpose and parent_request. Answers from INFRO's opt-in response cache carry a cache object and provider_billed: false, can be filtered with cache=hit, and their latency is reported apart from provider latency.
• Traffic changes watches the signals in the table above — a model answered by a new snapshot, output tokens, cached share, fallbacks, length-limited answers, p95 latency and time to first token, error codes — and raises each as a finding with the likely cause and the estimated change in provider cost. It reads metadata only, never prompts or answers.
• Request metadata is kept for a window set by your plan: 7 days on Developer, 14 on Startup, 30 on Team, 90 on Business. Prompt and completion capture is off by default; when an organization enables it, payloads are kept for a retention window it chooses, within its plan's limit. With guardrails (Business plan), a redacted prompt is what failover targets, capture and the OpenTelemetry export all see, and findings are counts per detector, never the matched text.
• From the Team plan, spans can be pushed to your own OpenTelemetry collector over OTLP/HTTP, nested in your trace when the request carries a W3C traceparent. Spans carry INFRO's own infro.* attributes and the core generative AI ones — gen_ai.operation.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, plus gen_ai.provider.name where the convention defines a value for that provider. Prompt and response captures are never exported.

For the people who will never open a request log, the console's Summary turns the same records into spend against budget and four status lights, with a weekly and monthly email; the format is described in the AI spend report leadership reads. See request traces, analytics and exports and OpenTelemetry for the full contract.