Can you replay production traffic to test a new model safely?

Replay is a gate, not a verdict

Replaying production traffic means taking requests you already served, sending them to a candidate model, and comparing what comes back with what the incumbent produced. It is the right tool when a retirement date is close and nobody has time to hand-label a few thousand examples: the prompts are real, the distribution is yours, and the only new work is scoring.

It is also open-loop. Nobody reacts to the candidate's answer, so replay cannot tell you whether users would accept it. It sees one turn at a time, so a model that drifts over a ten-turn conversation looks fine. And if a request triggered a tool with side effects, replaying it must not execute that tool again. Treat replay as the gate that decides whether the candidate earns a canary. The A/B testing post covers the live split that comes after, and the fallback equivalence post covers a narrower question: is the model good enough when it is only the backup.

The steps below are in order: capture, sample, scrub, replay, score, size, decide. If your deadline is short, the first step is the one that cannot be done late.

Capture: you can only replay what you logged

Prompt and response capture in INFRO is off until an owner or admin turns on request logging, and you choose whether to keep prompts, responses or both.¹ Retention is the smaller of the window you pick and your plan's maximum payload window: up to 24 hours on Developer, 14 days on Startup, 30 on Team and 90 on Business.¹ Content is never returned after its retention deadline, even if deletion has not yet run.² Metadata history is a separate window, so a request can still be listed after its payload is gone.³

That has a practical consequence. If logging is off today, turn it on today, record both halves, and start copying scrubbed samples into your own store daily. Reading content also needs payload access on the key, granted by an owner or admin.² If you log somewhere else, the same rule holds: the replay set is whatever your own logs kept.

Three details in the request record affect what you can use:

• content_truncated is true when the stored copy was shortened to fit. Drop those records; a truncated prompt is not the prompt the model saw.²
• Images, audio and video are kept as files beside the record and referenced as infro-file:<ref>, with download links valid for one hour. Fetch and attach them at sampling time, or exclude multimodal requests from this run.²
• Metadata such as feature, workflow and end-user id can be attached to the request, and you will use it to stratify the sample.⁴

Sample: stratify, then add the tails

A uniform random sample of a skewed workload over-tests the common case and under-tests the requests most likely to break. Stratify first, then sample within each stratum.

1. Pick strata that predict model behavior. Feature or workflow, prompt-size band, tools declared or not, images or not, streaming or not, language. The routing rule fields (feature, workflow, input_tokens, has_tools, has_images, stream) are a reasonable starting list because they are what you can already segment on.⁴
2. Set a floor per stratum. Allocate proportionally to traffic, but give every stratum a minimum (see the sizing section), and weight results back to traffic share when you report one headline number.
3. Add the tails on purpose. The request list can be sorted by slowest or costliest, and pages up to 100 at a time; the CSV export carries up to 50,000 rows per file.²³ Pull the top slice of each into a separate "tail" stratum and report it separately.
4. Deduplicate. Templated prompts repeat. Hash the normalized prompt and keep one per hash, or you will measure one prompt a thousand times.
5. Sample by session, not by request. Requests from one user or session are correlated. Draw the unit you will report on.
6. Keep only replayable requests. Exclude multi-turn requests whose earlier turns you cannot reproduce, requests whose tool results came from a live system, and anything truncated.

Scrub before the prompt leaves your boundary

Replaying sends production text to a candidate model, which may sit with a different provider under different retention terms than the incumbent. Check those terms first; the zero data retention post lists what to verify.

Then scrub. INFRO's guardrails can redact email addresses, phone numbers, card numbers, IBANs, US Social Security numbers, IP addresses and API keys, and match your own terms, before text reaches a provider.⁵ But the documentation is explicit that detection is pattern-based and can miss things: a name, a street address, a date of birth, or a number written out in words is not detected.⁵ For a replay set that will be stored and reviewed by people, add your own pass for names and addresses, or exclude strata that carry them.

Two rules keep the comparison honest:

• Scrub once, store the scrubbed text, and send that to both models. Redaction changes the prompt. If the candidate sees scrubbed text and you compare it to the logged answer to the original, you have measured the scrubbing, not the model.
• Re-run the incumbent on the scrubbed set as the control. This also gives you a fresh baseline from today's model snapshot rather than the one that answered weeks ago. Do this while the incumbent still answers: INFRO refuses a retired model from its shutdown date, naming the replacement, so a control run cannot wait until after it.

Replay: isolated, capped and tagged

Run replays on a key that cannot touch production. Each INFRO gateway key belongs to exactly one environment, so a key in a staging or replay environment never reaches a production connection. Put a spend limit on it: key limits always block, and a blocking limit reserves a worst case per request. When a chat request sets no max_tokens, that worst case assumes the model's maximum output (32,768 tokens where the catalog lists none), so set max_tokens explicitly in the harness to fit more concurrent requests under the cap.⁶

Tag every replayed request with a run id in metadata, and set logging: false on the request so INFRO does not capture the replayed payload; you store results in your own harness.¹ A route rule on a metadata key can send tagged requests to the candidate while the harness keeps one model name.⁴ The text endpoint takes the OpenAI request shape at https://api.infro.io/v1, so an ordinary OpenAI client is enough for the harness.⁷

import json, os
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

client = OpenAI(
    base_url="https://api.infro.io/v1",   # INFRO text endpoint, OpenAI-compatible
    api_key=os.environ["INFRO_REPLAY_KEY"],  # key in the replay environment, spend-limited
)

def run(item, model, arm, rep):
    r = client.chat.completions.create(
        model=model,
        messages=item["messages"],        # already scrubbed
        max_tokens=item["max_tokens"],    # always set; also bounds the budget reservation
        temperature=item.get("temperature", 1.0),
        tools=item.get("tools"),          # declared, but never executed here
        extra_body={"metadata": {"replay_run": "r-0142", "arm": arm},
                    "logging": False},
    )
    c = r.choices[0]
    return {"id": item["id"], "arm": arm, "rep": rep, "finish": c.finish_reason,
            "text": c.message.content, "tool_calls": [t.function.model_dump() for t in (c.message.tool_calls or [])],
            "out_tokens": r.usage.completion_tokens}

jobs = [(it, m, arm, rep) for it in json.load(open("sample.json"))
        for m, arm in [("INCUMBENT", "control"), ("CANDIDATE", "cand")]
        for rep in range(2)]           # two reps per arm to measure noise

with ThreadPoolExecutor(max_workers=8) as pool:  # keep concurrency under provider limits
    results = list(pool.map(lambda j: run(*j), jobs))

Never execute tool calls. Declare the tools, record the call the model chose and its arguments, and score that. Replay in the original streaming mode only if you are testing stream behavior; time to first token and per-token timing need streaming, while content scoring does not. The request log keeps per-request latency, token counts and estimated provider cost, so replayed traffic can be read back by tag from /v1/requests and /v1/usage for the cost and latency columns.² If you push spans to your own collector, INFRO's spans carry model, route, timing, token counts and estimated cost but not captured prompts or responses;³ the OpenTelemetry GenAI conventions now live in their own repository, so check there for current attribute names.⁸

Score without hand labels

With no labeled set, score in layers, cheapest and most objective first.

Layer 1: deterministic checks. These need no judge and catch the most common migration breakage:

• finish_reason distribution, especially the share cut off by length.
• JSON or schema validity, if you use structured outputs.
• Tool-call validity: the tool name exists, arguments parse and satisfy the schema, and the same tool is chosen as the control.
• Refusal rate and empty-answer rate.
• Output length ratio against the control, and language match.
• Latency and estimated cost per request.

Layer 2: pairwise judging against the control. Ask a judge model which of two answers better follows the original prompt, run it in both orders to cancel position bias, and count a disagreement between orders as a tie. Calibrate the judge on a small human-labeled slice before trusting it; the judge calibration post covers how. Pairwise judging needs no rubric per task, which is why it suits a deadline.

Layer 3: a noise floor. You ran two replications per arm. Measure how often the control disagrees with itself under the judge and the deterministic checks. The candidate has to differ from the control by no more than the control differs from itself, plus a margin you set in advance. Without this, sampling randomness gets reported as a model regression.

Layer 4: production signals you already have. If your product logs a downstream outcome (answer accepted, ticket reopened, thumbs), join it to the request id and check that your layer 1-2 scores predict it on the control arm. A score that does not predict outcomes is not worth a gate.

Layer 5: a human audit of disagreements only. Humans read the cases where candidate and control disagree and the judge was confident, plus a random slice of agreements. Fifty to a hundred reads is enough to catch a systematically wrong judge.

After cutover, INFRO's eval monitors can score a share of served chat answers in the background against your rubric and record the result per served model, which gives the same layer-2 signal on live traffic.

Size the sample

Two formulas cover most of it. For a pass rate such as schema validity, the number of requests for a 95% interval of half-width e around a rate p is about 1.96² × p(1 − p) / e². For a pairwise win rate near 50%, the half-width with n decisive comparisons (ties excluded) is about 0.98 / √n.

For a defect that should be absent, the rule of three applies: if you see zero failures in n independent requests, the 95% upper bound on the failure rate is about 3 / n. Zero malformed outputs in 300 requests bounds the rate near 1%.

Three adjustments matter more than the arithmetic. Size per stratum, because a stratum with 40 requests cannot clear a rare-failure bar however large the total. Count sessions or users as the unit when requests are correlated, since the effective sample is smaller than the row count. And set the margin before you look: decide the largest loss you would accept per stratum and per metric, write it down, then run.

Decide, document and hand off

Write the gate before the results exist. A reasonable shape: no stratum fails its deterministic check by more than the margin, the pairwise win-plus-tie rate against the control stays above your floor in every stratum, tail strata are reported separately, and latency and estimated cost are within bounds you agreed with finance. If a stratum fails, route only that stratum elsewhere. A routing rule that matches on a feature or metadata value can send it to a different target while the rest moves.⁴

Keep the run id, sample definition, scrub rules, prompts for the judge, thresholds and results together. NIST's AI RMF is voluntary guidance meant to help incorporate trustworthiness considerations into the evaluation and use of AI systems, and its Generative AI Profile, released in July 2024, is the companion for generative-specific risks;⁹ a dated, reproducible replay record is a concrete artifact to hang that on.

Then do what replay cannot: send a small live share to the candidate, watch the downstream signals, and keep the incumbent available until its shutdown date. The deprecation migration playbook lays out that sequence.