# Can you replay production traffic to test a new model safely?

URL: https://infro.io/blog/can-you-replay-production-traffic-to-test-a-new-model-safely
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-03
Last modified: 2026-11-03
Last reviewed: 2026-11-03 (next review due 2027-05-02)
Topic: Evaluation and models
Format: Methodology

> Yes, for single-turn, side-effect-free requests: sample logged requests by stratum, scrub them, re-run both the incumbent and the candidate on the scrubbed text, and score with deterministic checks, order-swapped pairwise judging and a noise floor. Replay is open-loop, so treat it as a gate before a canary, not a replacement for one.

Key takeaways:
- Replay works for single-turn, side-effect-free requests; it cannot judge multi-turn drift, tool side effects or user reaction, so follow it with a canary.
- You can only replay what you logged: payload capture is off by default and expires on a plan-defined window, so turn it on before you need it.
- Scrub, then re-run the incumbent on the scrubbed prompts as the control; never compare the candidate to the logged answer to an unscrubbed prompt.
- Score in layers: deterministic checks first, order-swapped pairwise judging second, a measured noise floor third, and a small human audit of disagreements last.
- Size the sample per stratum, not just overall: about 400 decisive comparisons gives roughly a five-point half-width at a 50% win rate.

## Replay is a gate, not a verdict

Replaying production traffic means taking requests you already served, sending them to a candidate model, and comparing what comes back with what the incumbent produced. It is the right tool when a retirement date is close and nobody has time to hand-label a few thousand examples: the prompts are real, the distribution is yours, and the only new work is scoring.

It is also open-loop. Nobody reacts to the candidate's answer, so replay cannot tell you whether users would accept it. It sees one turn at a time, so a model that drifts over a ten-turn conversation looks fine. And if a request triggered a tool with side effects, replaying it must not execute that tool again. Treat replay as the gate that decides whether the candidate earns a canary. The [A/B testing post](https://infro.io/blog/ab-testing-llm-models) covers the live split that comes after, and the [fallback equivalence post](https://infro.io/blog/is-your-fallback-model-good-enough-test-fallback-equivalence) covers a narrower question: is the model good enough when it is only the backup.

The steps below are in order: capture, sample, scrub, replay, score, size, decide. If your deadline is short, the first step is the one that cannot be done late.

## Capture: you can only replay what you logged

Prompt and response capture in INFRO is off until an owner or admin turns on request logging, and you choose whether to keep prompts, responses or both.[1] Retention is the smaller of the window you pick and your plan's maximum payload window: up to 24 hours on Developer, 14 days on Startup, 30 on Team and 90 on Business.[1] Content is never returned after its retention deadline, even if deletion has not yet run.[2] Metadata history is a separate window, so a request can still be listed after its payload is gone.[3]

That has a practical consequence. If logging is off today, turn it on today, record both halves, and start copying scrubbed samples into your own store daily. Reading content also needs payload access on the key, granted by an owner or admin.[2] If you log somewhere else, the same rule holds: the replay set is whatever your own logs kept.

Three details in the request record affect what you can use:

- `content_truncated` is true when the stored copy was shortened to fit. Drop those records; a truncated prompt is not the prompt the model saw.[2]
- Images, audio and video are kept as files beside the record and referenced as `infro-file:<ref>`, with download links valid for one hour. Fetch and attach them at sampling time, or exclude multimodal requests from this run.[2]
- Metadata such as feature, workflow and end-user id can be attached to the request, and you will use it to stratify the sample.[4]

## Sample: stratify, then add the tails

A uniform random sample of a skewed workload over-tests the common case and under-tests the requests most likely to break. Stratify first, then sample within each stratum.

1. **Pick strata that predict model behavior.** Feature or workflow, prompt-size band, tools declared or not, images or not, streaming or not, language. The routing rule fields (`feature`, `workflow`, `input_tokens`, `has_tools`, `has_images`, `stream`) are a reasonable starting list because they are what you can already segment on.[4]
2. **Set a floor per stratum.** Allocate proportionally to traffic, but give every stratum a minimum (see the sizing section), and weight results back to traffic share when you report one headline number.
3. **Add the tails on purpose.** The request list can be sorted by `slowest` or `costliest`, and pages up to 100 at a time; the CSV export carries up to 50,000 rows per file.[2][3] Pull the top slice of each into a separate "tail" stratum and report it separately.
4. **Deduplicate.** Templated prompts repeat. Hash the normalized prompt and keep one per hash, or you will measure one prompt a thousand times.
5. **Sample by session, not by request.** Requests from one user or session are correlated. Draw the unit you will report on.
6. **Keep only replayable requests.** Exclude multi-turn requests whose earlier turns you cannot reproduce, requests whose tool results came from a live system, and anything truncated.

## Scrub before the prompt leaves your boundary

Replaying sends production text to a candidate model, which may sit with a different provider under different retention terms than the incumbent. Check those terms first; the [zero data retention post](https://infro.io/blog/zero-data-retention-across-ai-providers-what-to-verify) lists what to verify.

Then scrub. INFRO's guardrails can redact email addresses, phone numbers, card numbers, IBANs, US Social Security numbers, IP addresses and API keys, and match your own terms, before text reaches a provider.[5] But the documentation is explicit that detection is pattern-based and can miss things: a name, a street address, a date of birth, or a number written out in words is not detected.[5] For a replay set that will be stored and reviewed by people, add your own pass for names and addresses, or exclude strata that carry them.

Two rules keep the comparison honest:

- **Scrub once, store the scrubbed text, and send that to both models.** Redaction changes the prompt. If the candidate sees scrubbed text and you compare it to the logged answer to the original, you have measured the scrubbing, not the model.
- **Re-run the incumbent on the scrubbed set as the control.** This also gives you a fresh baseline from today's model snapshot rather than the one that answered weeks ago. Do this while the incumbent still answers: INFRO refuses a retired model from its shutdown date, naming the replacement, so a control run cannot wait until after it.

## Replay: isolated, capped and tagged

Run replays on a key that cannot touch production. Each INFRO gateway key belongs to exactly one environment, so a key in a staging or replay environment never reaches a production connection. Put a spend limit on it: key limits always block, and a blocking limit reserves a worst case per request. When a chat request sets no `max_tokens`, that worst case assumes the model's maximum output (32,768 tokens where the catalog lists none), so set `max_tokens` explicitly in the harness to fit more concurrent requests under the cap.[6]

Tag every replayed request with a run id in metadata, and set `logging: false` on the request so INFRO does not capture the replayed payload; you store results in your own harness.[1] A route rule on a metadata key can send tagged requests to the candidate while the harness keeps one model name.[4] The text endpoint takes the OpenAI request shape at `https://api.infro.io/v1`, so an ordinary OpenAI client is enough for the harness.[7]

```python
import json, os
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",   # INFRO text endpoint, OpenAI-compatible
    api_key=os.environ["INFRO_REPLAY_KEY"],  # key in the replay environment, spend-limited
)

def run(item, model, arm, rep):
    r = client.chat.completions.create(
        model=model,
        messages=item["messages"],        # already scrubbed
        max_tokens=item["max_tokens"],    # always set; also bounds the budget reservation
        temperature=item.get("temperature", 1.0),
        tools=item.get("tools"),          # declared, but never executed here
        extra_body={"metadata": {"replay_run": "r-0142", "arm": arm},
                    "logging": False},
    )
    c = r.choices[0]
    return {"id": item["id"], "arm": arm, "rep": rep, "finish": c.finish_reason,
            "text": c.message.content, "tool_calls": [t.function.model_dump() for t in (c.message.tool_calls or [])],
            "out_tokens": r.usage.completion_tokens}

jobs = [(it, m, arm, rep) for it in json.load(open("sample.json"))
        for m, arm in [("INCUMBENT", "control"), ("CANDIDATE", "cand")]
        for rep in range(2)]           # two reps per arm to measure noise

with ThreadPoolExecutor(max_workers=8) as pool:  # keep concurrency under provider limits
    results = list(pool.map(lambda j: run(*j), jobs))
```

Never execute tool calls. Declare the tools, record the call the model chose and its arguments, and score that. Replay in the original streaming mode only if you are testing stream behavior; time to first token and per-token timing need streaming, while content scoring does not. The request log keeps per-request latency, token counts and estimated provider cost, so replayed traffic can be read back by tag from `/v1/requests` and `/v1/usage` for the cost and latency columns.[2] If you push spans to your own collector, INFRO's spans carry model, route, timing, token counts and estimated cost but not captured prompts or responses;[3] the OpenTelemetry GenAI conventions now live in their own repository, so check there for current attribute names.[8]

## Score without hand labels

With no labeled set, score in layers, cheapest and most objective first.

**Layer 1: deterministic checks.** These need no judge and catch the most common migration breakage:

- `finish_reason` distribution, especially the share cut off by `length`.
- JSON or schema validity, if you use structured outputs.
- Tool-call validity: the tool name exists, arguments parse and satisfy the schema, and the same tool is chosen as the control.
- Refusal rate and empty-answer rate.
- Output length ratio against the control, and language match.
- Latency and estimated cost per request.

**Layer 2: pairwise judging against the control.** Ask a judge model which of two answers better follows the original prompt, run it in both orders to cancel position bias, and count a disagreement between orders as a tie. Calibrate the judge on a small human-labeled slice before trusting it; the [judge calibration post](https://infro.io/blog/can-an-llm-judge-be-trusted-calibrating-graders-against-humans) covers how. Pairwise judging needs no rubric per task, which is why it suits a deadline.

**Layer 3: a noise floor.** You ran two replications per arm. Measure how often the control disagrees with itself under the judge and the deterministic checks. The candidate has to differ from the control by no more than the control differs from itself, plus a margin you set in advance. Without this, sampling randomness gets reported as a model regression.

**Layer 4: production signals you already have.** If your product logs a downstream outcome (answer accepted, ticket reopened, thumbs), join it to the request id and check that your layer 1-2 scores predict it on the control arm. A score that does not predict outcomes is not worth a gate.

**Layer 5: a human audit of disagreements only.** Humans read the cases where candidate and control disagree and the judge was confident, plus a random slice of agreements. Fifty to a hundred reads is enough to catch a systematically wrong judge.

After cutover, INFRO's eval monitors can score a share of served chat answers in the background against your rubric and record the result per served model, which gives the same layer-2 signal on live traffic.

## Size the sample

Two formulas cover most of it. For a pass rate such as schema validity, the number of requests for a 95% interval of half-width *e* around a rate *p* is about 1.96² × p(1 − p) / e². For a pairwise win rate near 50%, the half-width with *n* decisive comparisons (ties excluded) is about 0.98 / √n.

| Decisive comparisons | Half-width (points) |
| --- | --- |
| 100 | 9.8 |
| 200 | 6.9 |
| 400 | 4.9 |
| 1,000 | 3.1 |

Table: 95% half-width of a win rate near 50%, by number of decisive pairwise comparisons.

For a defect that should be absent, the rule of three applies: if you see zero failures in *n* independent requests, the 95% upper bound on the failure rate is about 3 / *n*. Zero malformed outputs in 300 requests bounds the rate near 1%.

> **Example: Worked sizing (illustrative numbers)**
> A support-reply workflow has five strata. Suppose you want the schema-validity rate known to within three points at a 90% pass rate: 1.96² × 0.9 × 0.1 / 0.03² is about 384 requests. You decide each stratum needs at least 150 decisive pairwise comparisons for a rough read (about 8 points) and the two largest need 400. Ties and order disagreements are often a third of pairs, so you sample about 1.5 times as many requests as decisive comparisons you want. Everything here is illustrative; plug in your own tie rate and tolerance.

Three adjustments matter more than the arithmetic. Size per stratum, because a stratum with 40 requests cannot clear a rare-failure bar however large the total. Count sessions or users as the unit when requests are correlated, since the effective sample is smaller than the row count. And set the margin before you look: decide the largest loss you would accept per stratum and per metric, write it down, then run.

## Decide, document and hand off

Write the gate before the results exist. A reasonable shape: no stratum fails its deterministic check by more than the margin, the pairwise win-plus-tie rate against the control stays above your floor in every stratum, tail strata are reported separately, and latency and estimated cost are within bounds you agreed with finance. If a stratum fails, route only that stratum elsewhere. A routing rule that matches on a feature or metadata value can send it to a different target while the rest moves.[4]

Keep the run id, sample definition, scrub rules, prompts for the judge, thresholds and results together. NIST's AI RMF is voluntary guidance meant to help incorporate trustworthiness considerations into the evaluation and use of AI systems, and its Generative AI Profile, released in July 2024, is the companion for generative-specific risks;[9] a dated, reproducible replay record is a concrete artifact to hang that on.

Then do what replay cannot: send a small live share to the candidate, watch the downstream signals, and keep the incumbent available until its shutdown date. The [deprecation migration playbook](https://infro.io/blog/model-deprecation-migration-playbook) lays out that sequence.

> **Note: When the deadline is days away**
> Turn on logging now. Sample 30 to 50 requests per stratum from what you have, run both arms with two reps, and gate on layers 1 and 3 plus a human read of every disagreement. Widen the sample while the canary runs.

## Questions

### Does a replay cost real money?

Yes. Replayed requests run on your own provider account and are billed by that provider at the rates in your contract. Estimate the run from the token counts of the sampled prompts and the candidate's rate, set max_tokens, and put a spend limit on the replay key so a bug cannot run past it.

### How long a window of traffic should the sample cover?

At least one full cycle of your traffic's weekly pattern, because weekday and weekend prompts often differ in topic, length and language. If your payload retention is shorter than that, sample on a schedule, for example daily, and accumulate the scrubbed set in your own store.

## Sources

1. Privacy & data — INFRO. https://infro.io/docs/privacy (accessed 2026-11-03)
2. Request traces — INFRO. https://infro.io/docs/observability/traces (accessed 2026-11-03)
3. Exports & OpenTelemetry — INFRO. https://infro.io/docs/observability/exports (accessed 2026-11-03)
4. Routes — INFRO. https://infro.io/docs/routes (accessed 2026-11-03)
5. Guardrails — INFRO. https://infro.io/docs/features/guardrails (accessed 2026-11-03)
6. Spend controls — INFRO. https://infro.io/docs/org/spend-controls (accessed 2026-11-03)
7. INFRO documentation overview — INFRO. https://infro.io/docs (accessed 2026-11-03)
8. Generative AI semantic conventions (moved) — OpenTelemetry. https://opentelemetry.io/docs/specs/semconv/gen-ai/ (accessed 2026-11-03)
9. AI Risk Management Framework — NIST. https://www.nist.gov/itl/ai-risk-management-framework (accessed 2026-11-03)