# LLM observability: what to measure in production

URL: https://infro.io/blog/llm-observability-production-guide
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-08-26
Last modified: 2026-10-13
Last reviewed: 2026-10-13 (next review due 2026-11-12)
Topic: Routing and reliability
Format: Guide

> LLM observability starts with one complete record per request: what was asked for and what served it, every failed attempt, time to first token and total duration, token counts, estimated cost and the outcome. Build the four golden signals on top of those records, add quality signals your application can compute without reading content, and keep prompt capture off unless a specific investigation needs it.

Key takeaways:
- The unit of LLM observability is the request record, not the log line: model requested and served, attempts, timing, tokens, cost and outcome together.
- A 200 status hides most LLM failures. Track malformed output, tool-call failures and fallbacks as first-class errors.
- Record the model the provider says answered, not only the one you asked for; a new snapshot behind an old name is a change worth an alert.
- Measure time to first token separately from total duration; averages hide the stalls users feel.
- Treat estimated cost as telemetry, recorded per request at the rate your account actually pays, with unknown rates recorded as unknown.
- Metadata answers most operational questions. Capture prompts only deliberately, with a retention window and access controls.

Teams often discover the limits of their AI monitoring the same way. A dashboard says error rates are flat and latency is normal, while support tickets say the assistant is producing nonsense, a batch job's bill has doubled, or answers take twenty seconds to start. Each of those is a failure an ordinary API dashboard cannot see: the status code was 200, the average latency was fine, and cost was not a metric at all.

This guide describes what to record for every LLM request, the signals to build on those records, how to connect them to the rest of your tracing, and how to do it without storing people's prompts.

## Why API monitoring misses LLM failures

Four properties of LLM traffic break the assumptions a generic HTTP dashboard makes:

- **Success is not correctness.** A response can arrive with status 200 and still fail its purpose: invalid JSON, a hallucinated tool argument, a refusal, an answer from a fallback model that was never evaluated for the task.
- **Latency has two shapes.** For streamed responses, the time until the first token decides whether a product feels responsive; the total duration decides throughput. One number cannot represent both.
- **Cost varies per request.** Two requests to the same endpoint can differ in cost by orders of magnitude, driven by prompt length, output length, cache hits and which account served them.
- **One request can be several attempts, and several calls.** When a gateway fails over, the successful response hides the failed attempts that preceded it — and their latency and, sometimes, their cost. A request can also trigger calls of its own: a classifier that picks the route, a safety check on the prompt, a background evaluation of the answer. Each costs money and can fail, and none of them is the request the user made.

## Adapt the four golden signals

Google's SRE guidance names four golden signals for monitoring a user-facing system: latency, traffic, errors and saturation[1]. Each needs an LLM-specific definition.

| Signal | For LLM traffic, measure | Why the plain version is not enough |
| --- | --- | --- |
| Latency | Time to first token and total duration, by model and target, as percentiles | Averages hide stalls; total duration hides slow starts |
| Traffic | Requests and tokens, input and output separately, by workload | Request counts do not predict cost or rate-limit pressure |
| Errors | Error statuses plus malformed output, tool-call failures, refusals and fallbacks | Most harmful failures return 200 |
| Saturation | Rate-limit responses per account and budget consumption per scope | The constraint is usually an allowance, not a CPU |

Table: The golden signals, redefined for model traffic.

## The request record is the unit of observability

A log line per request is easy to produce and hard to use. What operators need is one structured record per request that answers "what happened, why, and what did it cost" without joining three systems.

[Diagram: The fields a request record needs, grouped by the question each group answers. The field names shown are the ones INFRO uses; the grouping applies to any gateway.]

| Field group | Contents | The question it answers |
| --- | --- | --- |
| Identity | A request id returned to the caller, project, key, and your own metadata such as user, session or feature | Whose request was this, and where did it come from? |
| Path | Model requested, model served, route, provider, account, and each failed attempt with its error and latency | Where did it run, and did it recover from anything? |
| Outcome | HTTP status and the error type, code and message exactly as returned | Did the caller get what they needed? |
| Timing | Time to first token, total duration, and the upstream portion | Was it slow, and was the slowness ours or the provider's? |
| Usage and cost | Input and output tokens (or media units), estimated cost and where the rate came from | What did it consume, and what will the provider bill for it? |

Table: A request record that can answer an incident question on its own.

Two fields deserve emphasis. **Model requested versus model served** is how you detect a silent substitution — an alias that resolved differently, or a fallback that answered with another model. Record a third value too, the model id the provider's response names, because providers move a model name to a new snapshot without changing the name you send. **The request id** must be returned to the caller, so a support ticket, an application log and the gateway record can all be joined on one value.

Calls a request triggers belong in the same store, as rows of their own linked to the request they served, with their own cost and outcome. Folded into the parent, a classifier's timeouts and its share of spend disappear; the argument is made at length in [classifiers in the request path](https://infro.io/blog/classifiers-in-the-request-path).

## Connect LLM records to the rest of your traces

An LLM call is usually one span inside a larger request. W3C Trace Context standardizes the `traceparent` and `tracestate` headers that carry trace identity between services[2], so a gateway that accepts and propagates them lets a model call appear inside the trace of the user action that caused it.

OpenTelemetry is developing semantic conventions for generative AI that name the attributes such spans should carry — for example `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.provider.name`, `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`[3]. The conventions are marked as in development, so expect names to change and pin the version your collector expects.

## Treat cost as a telemetry signal

Model responses report usage: the Chat Completions response includes prompt and completion token counts, with cached prompt tokens broken out when prompt caching applied[4]. Anthropic reports cache writes and cache reads as separate fields, and prices them differently from ordinary input tokens[5]. Turning those counts into cost at the time of the request — rather than at the end of the month — makes cost a signal you can alert on and attribute.

Three rules keep it trustworthy:

1. **Use the rate your account pays.** A negotiated rate, or a cloud platform's rate card, can differ from the model vendor's list price. Record which rate was used.
2. **Record unknown as unknown.** A model with no rate on file should produce an empty cost, not zero. Zero reads as free and quietly understates spend.
3. **Keep the estimate separate from the invoice.** A per-request estimate is what budgets and attribution are computed from; the provider's invoice is the authority. Compare them periodically and investigate a sustained gap.

If anything in front of the provider answers requests itself — an exact-match response cache, for instance — mark those responses. A cache hit costs the provider nothing and returns in milliseconds; mixed into the same series, it flatters both p95 latency and cost per request, and hides a provider getting slower. Report hits against lookups, and their latency apart from provider latency. When to use such a cache at all is covered in [exact-match response caching](https://infro.io/blog/llm-response-cache-exact-match).

## Quality signals that do not require reading prompts

Most quality regressions can be detected from signals the application computes anyway:

- the share of responses that fail schema validation or parsing;
- tool calls with invalid arguments, and tools that returned errors;
- refusals and truncated outputs (a finish reason of length rather than stop);
- retries triggered by the application itself;
- user corrections, regenerations and escalations to a person;
- scores from a sampled, access-controlled evaluation run.

Record these as metadata on the request or as events joined by request id. They catch the release that made a prompt worse without anyone reading a customer's conversation.

## Changes that arrive with status 200

Some regressions are not caused by your release at all. A provider moves a model name to a new snapshot and answers get longer. A small edit near the top of a prompt stops the provider's prompt cache from matching, and input cost rises. A primary starts timing out for a few requests, and they quietly move to a fallback with a higher rate. None of these produces an error, and each is visible in metadata alone:

| Signal | What usually caused it |
| --- | --- |
| A served model id not seen in recent weeks | The provider moved the model name to a new snapshot |
| Output tokens per request rising | A prompt or parameter change, or a new snapshot |
| Cached share of prompt tokens falling | The start of the prompt changed |
| Share of requests served by a fallback rising | The primary is failing or slow for some requests |
| Share of answers stopped by the length limit rising | Answers are reaching `max_tokens` |

Table: Silent changes detectable from request records, without reading content.

Compare each against its own recent baseline, per model and per account, and require both a relative and an absolute change before alerting, or every quiet night becomes a finding. What metadata cannot tell you is whether answers got worse; that needs an evaluation, as described in [how to evaluate LLMs on real workloads](https://infro.io/blog/evaluate-llms-real-workloads).

## Metrics that should change a decision

A metric earns a place on a dashboard when crossing a threshold triggers an action someone owns.

| Metric | Action when it moves |
| --- | --- |
| Fallback rate per account | Investigate the account; consider taking it out of rotation |
| p95 time to first token per model and target | Check the target's health; lower the per-attempt timeout; move interactive traffic |
| Malformed-output rate after a release | Roll back the prompt or model change |
| Rate-limit responses per account | Spread load across accounts or request a higher allowance |
| Cost per successful task per workload | Re-evaluate the model choice or reduce context |
| Unpriced request share | Add the missing rates so budgets and attribution are complete |
| New served model id on a connection | Re-run the evaluation for workloads on that model |

Table: Each metric paired with the decision it exists to drive.

## Privacy, retention and access

Request metadata can reveal a great deal — which customers use a feature, when, and how heavily — and prompts and completions can contain exactly the personal and confidential data that sensitive-information-disclosure risks describe[6]. Decide the policy before the first incident, when there is time to think:

- Keep prompt and completion capture off by default. Enable it for a defined purpose, a defined scope and a defined retention window.
- Keep operational metadata longer than content. Most investigations need the path, timing and error, not the text.
- Redact secrets before anything is logged, including inside tool arguments, and make sure the redacted text is what every downstream copy sees: retries, captures and exported spans.
- Apply the same role boundaries to reading request data as to managing production keys.

## An annotated request record

The record below is illustrative and abbreviated, in the shape INFRO returns. Each field answers one of the questions above.

```json
{
  "id": "req_…",
  "status": "ok",
  "http_status": 200,
  "model_requested": "support-drafts",
  "model": "anthropic/claude-sonnet-5",
  "served_model": "claude-sonnet-5",
  "finish_reason": "stop",
  "route": "primary",
  "route_slug": "support-drafts",
  "provider": "anthropic",
  "connection_name": "Anthropic production",
  "fallback_attempts": [],
  "latency_ms": 4210,
  "first_token_ms": 640,
  "usage": {
    "prompt_tokens": 1830,
    "completion_tokens": 412,
    "total_tokens": 2242,
    "cost": 0.01167,
    "price_source": "list"
  },
  "metadata": { "user_id": "cust_8231", "tags": { "feature": "ticket-reply" } }
}
```

- `model_requested` is a route slug and `model` is what served it, so a substitution would be visible here.
- `served_model` is the id the provider's response named; a new value here, with the same `model`, is a snapshot change.
- `finish_reason` is `stop`, not `length`, so the answer was not cut off.
- `fallback_attempts` is empty: nothing failed before this response.
- `first_token_ms` and `latency_ms` separate a slow start from a long answer.
- `price_source` says the estimate used the vendor's list rate, not a negotiated one.
- `metadata` carries attribution you supplied, without any prompt content.

## How INFRO records requests

Every request through INFRO produces a record like the one above, including refused and failed requests once the gateway knows which model was asked for. Every response carries an `X-INFRO-Request-Id` header with the same id. The records are about traffic on your own provider accounts; every cost in them is an estimate of what your provider will bill, `null` when the model is unpriced, and never includes INFRO's subscription.

- `GET /v1/requests` lists records with filters for status, model, project, key, provider, connection, route, and the `user_id`, `session_id` and `tag.<key>` values you send in request `metadata`; `sort` can be `newest`, `slowest` or `costliest`. `GET /v1/requests/{id}` returns one record with its timing spans.
- `GET /v1/usage` aggregates requests, units, estimated cost, unpriced requests, error rate and p50, p95 and p99 latency by hour, day or month, grouped by model, modality, project, key, endpoint, provider, connection or route. Latency is measured on calls that reached your provider.
- Calls INFRO makes for a request — a guardrail classifier check, a route's prompt classifier, an eval monitor's score — are rows of their own with their own estimated cost, carrying `purpose` and `parent_request`. Answers from INFRO's opt-in [response cache](https://infro.io/docs/features/caching) carry a `cache` object and `provider_billed: false`, can be filtered with `cache=hit`, and their latency is reported apart from provider latency.
- [Traffic changes](https://infro.io/docs/observability/traffic-changes) watches the signals in the table above — a model answered by a new snapshot, output tokens, cached share, fallbacks, length-limited answers, p95 latency and time to first token, error codes — and raises each as a finding with the likely cause and the estimated change in provider cost. It reads metadata only, never prompts or answers.
- Request metadata is kept for a window set by your plan: 7 days on Developer, 14 on Startup, 30 on Team, 90 on Business. Prompt and completion capture is off by default; when an organization enables it, payloads are kept for a retention window it chooses, within its plan's limit. With [guardrails](https://infro.io/docs/features/guardrails) (Business plan), a redacted prompt is what failover targets, capture and the OpenTelemetry export all see, and findings are counts per detector, never the matched text.
- From the Team plan, spans can be pushed to your own OpenTelemetry collector over OTLP/HTTP, nested in your trace when the request carries a W3C `traceparent`. Spans carry INFRO's own `infro.*` attributes and the core generative AI ones — `gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`, plus `gen_ai.provider.name` where the convention defines a value for that provider. Prompt and response captures are never exported.

For the people who will never open a request log, the console's Summary turns the same records into spend against budget and four status lights, with a weekly and monthly email; the format is described in [the AI spend report leadership reads](https://infro.io/blog/ai-spend-report-for-leadership). See [request traces](https://infro.io/docs/observability/traces), [analytics](https://infro.io/docs/observability/analytics) and [exports and OpenTelemetry](https://infro.io/docs/observability/exports) for the full contract.

## Sources

1. Monitoring Distributed Systems — Site Reliability Engineering — Google. https://sre.google/sre-book/monitoring-distributed-systems/ (accessed 2026-10-13)
2. Trace Context (W3C Recommendation) — W3C. https://www.w3.org/TR/trace-context/ (accessed 2026-10-13)
3. OpenTelemetry semantic conventions for generative AI — OpenTelemetry. https://github.com/open-telemetry/semantic-conventions-genai (accessed 2026-10-13)
4. Create chat completion — API reference — OpenAI. https://developers.openai.com/api/reference/go/resources/chat/subresources/completions/methods/create (accessed 2026-10-13)
5. Prompt caching — Anthropic. https://platform.claude.com/docs/en/build-with-claude/prompt-caching (accessed 2026-10-13)
6. OWASP Top 10 for LLM Applications 2025 — OWASP GenAI Security Project. https://genai.owasp.org/llm-top-10/ (accessed 2026-10-13)

## Revision history

- 2026-10-13: Reviewed against every source. Added detection of changes that return 200 (a new model snapshot behind an old name, longer answers, a falling cached share), recording the extra calls a request triggers — classifier checks, evaluations — as rows of their own, and keeping response-cache hits apart from provider latency and cost. Corrected the INFRO section: exported spans now carry the core OpenTelemetry generative AI attributes alongside INFRO's own, and it now covers traffic-change findings, guardrail redaction before capture and export, and the leadership Summary. Updated the OpenAI API reference link, which had moved.
- 2026-09-11: Expanded from a short overview into a full guide: the request record as the unit of observability, golden signals adapted to LLM traffic, tracing and OpenTelemetry conventions, cost as telemetry, quality signals without content, privacy and retention, and an annotated record. Added cited sources.