# Catch silent model drift with a golden set and canary prompts

URL: https://infro.io/blog/catch-silent-model-drift-with-a-golden-set-and-canary-prompts
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-14
Last modified: 2026-11-14
Last reviewed: 2026-11-14 (next review due 2027-05-13)
Topic: Evaluation and models
Format: Methodology

> Run a fixed golden set of 40–100 prompts through the same gateway route as production on a schedule, record the model the response names, and alert when the score falls outside a noise band measured from your own baseline. Agree the rollback rule before the first alert, and pair the score with metadata signals such as served model, output length and fallbacks.

Key takeaways:
- A pinned id is a promise about a name, not about behavior: a vendor can route a stable id to a successor on deprecation, and your own route can move traffic to a fallback.
- A golden set of 40–100 prompts with mostly deterministic checks, sampled several times per run, is enough to see drift that metadata cannot show.
- Set alert thresholds from your own baseline noise, and require both a statistical and a practical drop before anything pages a person.
- Combine the quality score with the served model id, output length, truncation and fallback share: each alone is ambiguous, together they usually name the cause.
- Write the rollback rule before the first alert, including what you do when the old snapshot is no longer served.

## Why a pinned id still drifts

Pinning protects you from upgrades the vendor announces by name. It does not protect you from three other things. First, a stable id can be rerouted: on 8 October 2026 Google began answering requests for `gemini-3.7-flash` with `gemini-3.8-flash`, and xAI sends `grok-voice-transcribe-1.0` requests to 2.0 since its end of life [1]. Second, your own routing can change what answers: a fallback target, a rule that sends some requests elsewhere, an auto-routed alias. Third, the same model id can report differently on different providers, so a failover changes the serving path without changing the name you sent.

None of these raises an error. INFRO's traffic-changes docs describe the same class of problem: a model name moves to a new snapshot and answers get longer, or a primary times out for a few requests and they quietly move to a fallback [2]. Metadata catches the shape of that. It cannot tell you the answers got worse, because the gateway reads request metadata, not prompts or answers [2]. For that you need a fixed set of questions with known-good outcomes, asked repeatedly, through the same path production uses.

This post covers that second half: the golden set, the canary schedule, the thresholds, and the rollback rule. For the naming conventions and pinning rules, see [model aliases vs pinned snapshots](https://infro.io/blog/model-aliases-vs-pinned-snapshots).

## Build a golden set that fits in one afternoon

A golden set is a small, frozen list of inputs with a machine-checkable definition of "good". Start with 40 to 100 items. Fewer than 40 and one flaky item moves the score by a visible amount; more than 100 and you will stop maintaining it.

OpenAI's evals guide describes the same minimal shape: a schema for each test item and testing criteria such as a string check against a human-supplied label [3]. That platform is being retired: it became read-only on October 31, 2026 and is scheduled to shut down on November 30, 2026 [3]. Keep your golden set in your own repository as plain JSONL so it outlives any vendor's eval tool; see [export your evals and rebuild them in CI](https://infro.io/blog/openai-evals-shutdown-migration).

Draw items from four places:

- Real production prompts, scrubbed of personal data, stratified by feature and by input length.
- Format-critical prompts: anything whose output a parser consumes (JSON, enums, tool-call arguments).
- Boundary prompts: requests that sit near a refusal line or a policy rule, where a changed safety profile shows first.
- Past incidents: every prompt that once produced a bad answer in production, with the fix as the expected outcome.

Prefer deterministic checks (exact label, JSON that parses and has the required keys, a regex, a length bound, a required citation) for at least 70% of items. Use a model judge only for the remainder, and calibrate it first; [calibrating graders against humans](https://infro.io/blog/can-an-llm-judge-be-trusted-calibrating-graders-against-humans) covers that.

> **Example: Illustrative golden set of 60 items**
> These counts are made up to show the proportions, not taken from any real system.
> 
> | Stratum | Items | Check type |
> | --- | --- | --- |
> | Core task, short input | 20 | Exact label or regex |
> | Core task, long input | 10 | Label plus length bound |
> | Structured output | 12 | JSON parses, keys present, enum valid |
> | Refusal boundary | 8 | Refuses or answers, as labelled |
> | Past incidents | 6 | Per-incident assertion |
> | Judge-scored free text | 4 | Rubric score, calibrated judge |

Version the file. Every change to the set resets the baseline, so record the set's hash next to each result and never compare scores across two hashes.

## Run canaries through the gateway, not around it

A canary that calls the provider directly measures the provider. A canary that goes through your gateway, on the route production uses, measures what your users get, including routing rules, fallbacks and parameter handling. The gateway's chat endpoint takes either a route slug or a `vendor/model-name` id in the `model` field [4], so the canary can call the exact route name production calls.

Four details make the result trustworthy:

1. **Dedicated key and project.** Give the canary its own gateway key in the same environment as production so its cost and requests are separable in usage and the request log. The request log records the model, the connection that served it, outcome, timings, usage and estimated cost, and can be queried per key [5].
2. **No response cache on the canary path.** An exact-match cache answers an identical request without reaching the provider, and cache hits are excluded from traffic-change signals for that reason [2]. A cached canary would score your cache. Keep caching off for the canary key or route.
3. **Sample each item several times.** Sampling parameters are not always honored: some current models refuse temperature, top_p and top_k, and the gateway drops them and lists what it dropped in the `X-INFRO-Params-Dropped` header; `seed` is best-effort [4]. Run each item three to five times and store the per-item pass rate.
4. **Record what answered.** Store the model id in the response body, the finish reason, completion tokens, latency and the dropped-params header with every sample. The model id is the first evidence of a swap [1].

```python
import json, time, re
from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1", api_key=CANARY_KEY)
ROUTE = "support-triage"   # the same route slug production calls
K = 3

def check(item, text):
    kind = item["check"]
    if kind == "exact":
        return text.strip() == item["expected"]
    if kind == "regex":
        return re.search(item["expected"], text) is not None
    if kind == "json_keys":
        try:
            obj = json.loads(text)
        except ValueError:
            return False
        return all(k in obj for k in item["expected"])
    return False

def run_item(item):
    samples = []
    for _ in range(K):
        t0 = time.time()
        raw = client.chat.completions.with_raw_response.create(
            model=ROUTE,
            messages=item["messages"],
            max_tokens=item.get("max_tokens", 400),
            temperature=0,
        )
        resp = raw.parse()
        text = resp.choices[0].message.content or ""
        samples.append({
            "ok": check(item, text),
            "served_model": resp.model,
            "finish": resp.choices[0].finish_reason,
            "out_tokens": resp.usage.completion_tokens,
            "latency_s": round(time.time() - t0, 3),
            "dropped": raw.headers.get("X-INFRO-Params-Dropped"),
        })
    return {"id": item["id"], "stratum": item["stratum"], "samples": samples}
```

Schedule it from cron or your CI scheduler, append one JSON line per run to storage you control, and compute the run score as the mean of per-item pass rates. Cap the canary key with a budget so a looping scheduler cannot spend without limit.

> **Example: Illustrative canary volume**
> 60 items × 3 samples × 24 runs a day = 4,320 calls a day. At an assumed 1,000 tokens per call that is about 4.3 million tokens a day, which you multiply by your own provider's rate for the pinned model. These figures are illustrative; compute yours before choosing the cadence.

## Set thresholds from your own noise

A fixed rule such as "alert below 90%" fails twice: it fires constantly on a noisy set and never on a stable set that slides from 99% to 91%. Measure the noise first.

Run the full canary for at least 14 scheduled runs with nothing changed. Take the mean and standard deviation of the run score, per stratum as well as overall. Then alert only when the drop is both statistically large and practically large:

```python
import statistics

def drifted(history, today, min_drop=0.05):
    base = history[-14:]
    mu = statistics.mean(base)
    sd = max(statistics.stdev(base), 0.01)
    return today < mu - max(3 * sd, min_drop)
```

The `min_drop` floor stops a very quiet baseline from paging on a one-item wobble. Apply the rule per stratum: a six-point drop confined to the structured-output stratum is a real signal that an overall mean would hide. For format-critical strata, add a hard floor instead of a band: if a parser consumes the output, a pass rate below your own tolerance is an incident whatever the history says.

Pair the score with the metadata signals that need no judge. INFRO's traffic-changes docs give workable reference points: output tokens per request up at least 15% and 20 tokens, `finish_reason: length` share up 3 percentage points, fallback share up 5 points, p95 latency up 25% and 300 ms two days running [2]. Those are the gateway's own thresholds; take them as a starting point and tune against your canary's variance.

| Signal | Source | Starting threshold |
| --- | --- | --- |
| Run score, per stratum | Canary | Below baseline mean minus the larger of 3 SD or 5 points |
| Served model id | Canary and request log | Any id not seen for this route in the last 14 days |
| Output tokens per sample | Canary | Up 15% over baseline median |
| Truncated samples | Canary | Up 3 points |
| Parse failures | Canary | Any rise above your tolerance |

Table: canary alert thresholds. The thresholds are proposals to tune, not measured effects.

## Read the signals together

Each signal is ambiguous alone, and the combination usually names the cause.

| Score | Served model id | Length or tokens | What to do first |
| --- | --- | --- | --- |
| Down | New | Changed | Treat as a snapshot swap. Run the full evaluation on the new id before deciding. |
| Down | Same | Same | Rerun twice. If it persists, check the prompt, the route and your own deploys. |
| Down | Same | Longer, more truncation | Likely a parameter or reasoning-effort change. Check the dropped-params header and `max_tokens`. |
| Same | New | Changed | Quality held on this set. Widen the set before trusting it, and re-estimate cost. |
| Down only on fallback samples | Different on those | Any | Your fallback is the weak link. See [fallback equivalence](https://infro.io/blog/is-your-fallback-model-good-enough-test-fallback-equivalence). |

A caveat on the served-model column: a gateway can only see a new id when the provider's reply reports one. A provider that echoes the requested name while serving something else leaves that signal blank, and only the derived metrics and your score remain [1]. This is the reason the golden set exists.

Ship canary results as telemetry, not just as log lines. INFRO can push request-path spans to your own collector, and its docs list exports for the request log and usage series [6]. OpenTelemetry's GenAI semantic conventions define spans, metrics and events for these calls, but they have moved to their own repository, so read attribute names from there rather than from memory [7]. Emit the run score and stratum scores as metrics, and route alerts to the on-call channel your team already uses.

## Decide on rollback before the first alert

The worst time to argue about thresholds is during the alert. Write the rule down, and review it as you would any other control; NIST's AI RMF is voluntary guidance, and its Generative AI Profile (NIST-AI-600-1, released July 26, 2024) proposes risk-management actions for generative AI that an organization can map its own monitoring to [8].

A rule that works in practice has three tiers:

1. **Watch.** One run outside the band. Rerun immediately, double the sample count, and do nothing else. Most alerts end here.
2. **Confirm.** Two consecutive runs outside the band, or one run outside plus a new served model id. Open an incident, diff failing items against the baseline, and run your larger evaluation on the old and new behavior side by side. Replaying production traffic is one way to get that sample; see [replaying production traffic safely](https://infro.io/blog/can-you-replay-production-traffic-to-test-a-new-model-safely).
3. **Roll back.** Confirmed drop in a critical stratum (format, safety boundary, a past-incident item), or an overall drop beyond your product's tolerance, and the larger evaluation agrees.

Rolling back is only possible if the old behavior is still reachable, and after a vendor reroute it often is not. Your options, in order: point the route at another platform that still serves the old id (the same model can have different lifecycle dates on different platforms, as Google's Gemini API and Vertex AI did for 3.7 Flash [1]); move to a different pinned snapshot you have already evaluated; or keep the new model and adapt the prompt, then re-baseline. With a route, the move is a configuration change rather than a deploy; INFRO does not answer with a different model on your behalf, so the decision stays with you [9].

After any rollback or deliberate model change, freeze the old baseline, start a new one, and keep both scores in the incident record.

## What the gateway can and cannot do here

INFRO is a BYOK control plane: your providers bill you directly, and the canary runs on your own connections. Three shipped parts apply here. Traffic changes raises findings for a new served snapshot, longer answers, truncation, more fallbacks and slower responses, each with a likely cause [2]; it does not judge answer quality. Eval monitors score a share of served chat answers against your rubric in the background and summarize per served model, which complements the scheduled canary with a sample of live traffic. And findings can be delivered to email, Slack, signed webhooks, Teams or PagerDuty. The golden set, thresholds and rollback rule remain yours to own.

## Questions

### How often should canaries run?

Match the cadence to how fast you need to know and what a run costs. Daily is a sensible start for most features, hourly for revenue-critical paths. Provider notices such as a deprecation or a new snapshot are a reason to trigger an extra run on the same day, outside the schedule.

### Does temperature 0 make canary results repeatable?

Not reliably. Some models refuse sampling parameters and the gateway drops them, and seed is best-effort on providers that support it [4]. Treat every score as a sample from a distribution, repeat each item, and measure the noise band before setting thresholds.

### Can a gateway tell me the answers got worse?

A gateway that reads only request metadata cannot; it can report that the served model, length, latency or fallback share changed [2]. Quality needs an evaluation that reads answers: your golden set, or a background scorer against your rubric.

## Sources

1. Model aliases vs pinned snapshots: avoid silent model swaps — INFRO. https://infro.io/blog/model-aliases-vs-pinned-snapshots (accessed 2026-11-14)
2. Traffic changes — Observability — INFRO Docs. https://infro.io/docs/observability/traffic-changes (accessed 2026-11-14)
3. Working with evals — OpenAI. https://developers.openai.com/api/docs/guides/evals.md (accessed 2026-11-14)
4. Chat completions — Modalities — INFRO Docs. https://infro.io/docs/api/chat-completions (accessed 2026-11-14)
5. Request traces — Observability — INFRO Docs. https://infro.io/docs/observability/traces (accessed 2026-11-14)
6. INFRO documentation overview — INFRO Docs. https://infro.io/docs (accessed 2026-11-14)
7. Generative AI semantic conventions (moved) — OpenTelemetry. https://opentelemetry.io/docs/specs/semconv/gen-ai/index.md (accessed 2026-11-14)
8. AI Risk Management Framework — NIST. https://www.nist.gov/itl/ai-risk-management-framework (accessed 2026-11-14)
9. Manage AI model retirements and silent changes — INFRO. https://infro.io/use-cases/model-change-management (accessed 2026-11-14)
