Skip to content

Evaluation and modelsMethodology

Catch silent model drift with a golden set and canary prompts

Pinning a model id does not stop answers from changing. This method builds a small golden set, runs it on a schedule through the same route production uses, and turns the scores into alerts with a pre-agreed rollback rule.

By INFRO Editorial

Published

9 min read

A single amber path on a dark ground forking into two glass branches of slightly different brightness

In short

Run a fixed golden set of 40–100 prompts through the same gateway route as production on a schedule, record the model the response names, and alert when the score falls outside a noise band measured from your own baseline. Agree the rollback rule before the first alert, and pair the score with metadata signals such as served model, output length and fallbacks.

Key takeaways

  • A pinned id is a promise about a name, not about behavior: a vendor can route a stable id to a successor on deprecation, and your own route can move traffic to a fallback.
  • A golden set of 40–100 prompts with mostly deterministic checks, sampled several times per run, is enough to see drift that metadata cannot show.
  • Set alert thresholds from your own baseline noise, and require both a statistical and a practical drop before anything pages a person.
  • Combine the quality score with the served model id, output length, truncation and fallback share: each alone is ambiguous, together they usually name the cause.
  • Write the rollback rule before the first alert, including what you do when the old snapshot is no longer served.

Why a pinned id still drifts

Pinning protects you from upgrades the vendor announces by name. It does not protect you from three other things. First, a stable id can be rerouted: on 8 October 2026 Google began answering requests for gemini-3.7-flash with gemini-3.8-flash, and xAI sends grok-voice-transcribe-1.0 requests to 2.0 since its end of life 1. Second, your own routing can change what answers: a fallback target, a rule that sends some requests elsewhere, an auto-routed alias. Third, the same model id can report differently on different providers, so a failover changes the serving path without changing the name you sent.

None of these raises an error. INFRO's traffic-changes docs describe the same class of problem: a model name moves to a new snapshot and answers get longer, or a primary times out for a few requests and they quietly move to a fallback 2. Metadata catches the shape of that. It cannot tell you the answers got worse, because the gateway reads request metadata, not prompts or answers 2. For that you need a fixed set of questions with known-good outcomes, asked repeatedly, through the same path production uses.

This post covers that second half: the golden set, the canary schedule, the thresholds, and the rollback rule. For the naming conventions and pinning rules, see model aliases vs pinned snapshots.

Build a golden set that fits in one afternoon

A golden set is a small, frozen list of inputs with a machine-checkable definition of "good". Start with 40 to 100 items. Fewer than 40 and one flaky item moves the score by a visible amount; more than 100 and you will stop maintaining it.

OpenAI's evals guide describes the same minimal shape: a schema for each test item and testing criteria such as a string check against a human-supplied label 3. That platform is being retired: it became read-only on October 31, 2026 and is scheduled to shut down on November 30, 2026 3. Keep your golden set in your own repository as plain JSONL so it outlives any vendor's eval tool; see export your evals and rebuild them in CI.

Draw items from four places:

  • Real production prompts, scrubbed of personal data, stratified by feature and by input length.
  • Format-critical prompts: anything whose output a parser consumes (JSON, enums, tool-call arguments).
  • Boundary prompts: requests that sit near a refusal line or a policy rule, where a changed safety profile shows first.
  • Past incidents: every prompt that once produced a bad answer in production, with the fix as the expected outcome.

Prefer deterministic checks (exact label, JSON that parses and has the required keys, a regex, a length bound, a required citation) for at least 70% of items. Use a model judge only for the remainder, and calibrate it first; calibrating graders against humans covers that.

Version the file. Every change to the set resets the baseline, so record the set's hash next to each result and never compare scores across two hashes.

Run canaries through the gateway, not around it

A canary that calls the provider directly measures the provider. A canary that goes through your gateway, on the route production uses, measures what your users get, including routing rules, fallbacks and parameter handling. The gateway's chat endpoint takes either a route slug or a vendor/model-name id in the model field 4, so the canary can call the exact route name production calls.

Four details make the result trustworthy:

  1. Dedicated key and project. Give the canary its own gateway key in the same environment as production so its cost and requests are separable in usage and the request log. The request log records the model, the connection that served it, outcome, timings, usage and estimated cost, and can be queried per key 5.
  2. No response cache on the canary path. An exact-match cache answers an identical request without reaching the provider, and cache hits are excluded from traffic-change signals for that reason 2. A cached canary would score your cache. Keep caching off for the canary key or route.
  3. Sample each item several times. Sampling parameters are not always honored: some current models refuse temperature, top_p and top_k, and the gateway drops them and lists what it dropped in the X-INFRO-Params-Dropped header; seed is best-effort 4. Run each item three to five times and store the per-item pass rate.
  4. Record what answered. Store the model id in the response body, the finish reason, completion tokens, latency and the dropped-params header with every sample. The model id is the first evidence of a swap 1.
canary.py
import json, time, re
from openai import OpenAI

client = OpenAI(base_url="https://api.infro.io/v1", api_key=CANARY_KEY)
ROUTE = "support-triage"   # the same route slug production calls
K = 3

def check(item, text):
    kind = item["check"]
    if kind == "exact":
        return text.strip() == item["expected"]
    if kind == "regex":
        return re.search(item["expected"], text) is not None
    if kind == "json_keys":
        try:
            obj = json.loads(text)
        except ValueError:
            return False
        return all(k in obj for k in item["expected"])
    return False

def run_item(item):
    samples = []
    for _ in range(K):
        t0 = time.time()
        raw = client.chat.completions.with_raw_response.create(
            model=ROUTE,
            messages=item["messages"],
            max_tokens=item.get("max_tokens", 400),
            temperature=0,
        )
        resp = raw.parse()
        text = resp.choices[0].message.content or ""
        samples.append({
            "ok": check(item, text),
            "served_model": resp.model,
            "finish": resp.choices[0].finish_reason,
            "out_tokens": resp.usage.completion_tokens,
            "latency_s": round(time.time() - t0, 3),
            "dropped": raw.headers.get("X-INFRO-Params-Dropped"),
        })
    return {"id": item["id"], "stratum": item["stratum"], "samples": samples}

Schedule it from cron or your CI scheduler, append one JSON line per run to storage you control, and compute the run score as the mean of per-item pass rates. Cap the canary key with a budget so a looping scheduler cannot spend without limit.

Set thresholds from your own noise

A fixed rule such as "alert below 90%" fails twice: it fires constantly on a noisy set and never on a stable set that slides from 99% to 91%. Measure the noise first.

Run the full canary for at least 14 scheduled runs with nothing changed. Take the mean and standard deviation of the run score, per stratum as well as overall. Then alert only when the drop is both statistically large and practically large:

verdict.py
import statistics

def drifted(history, today, min_drop=0.05):
    base = history[-14:]
    mu = statistics.mean(base)
    sd = max(statistics.stdev(base), 0.01)
    return today < mu - max(3 * sd, min_drop)

The min_drop floor stops a very quiet baseline from paging on a one-item wobble. Apply the rule per stratum: a six-point drop confined to the structured-output stratum is a real signal that an overall mean would hide. For format-critical strata, add a hard floor instead of a band: if a parser consumes the output, a pass rate below your own tolerance is an incident whatever the history says.

Pair the score with the metadata signals that need no judge. INFRO's traffic-changes docs give workable reference points: output tokens per request up at least 15% and 20 tokens, finish_reason: length share up 3 percentage points, fallback share up 5 points, p95 latency up 25% and 300 ms two days running 2. Those are the gateway's own thresholds; take them as a starting point and tune against your canary's variance.

SignalSourceStarting threshold
Run score, per stratumCanaryBelow baseline mean minus the larger of 3 SD or 5 points
Served model idCanary and request logAny id not seen for this route in the last 14 days
Output tokens per sampleCanaryUp 15% over baseline median
Truncated samplesCanaryUp 3 points
Parse failuresCanaryAny rise above your tolerance
canary alert thresholds. The thresholds are proposals to tune, not measured effects.

Read the signals together

Each signal is ambiguous alone, and the combination usually names the cause.

ScoreServed model idLength or tokensWhat to do first
DownNewChangedTreat as a snapshot swap. Run the full evaluation on the new id before deciding.
DownSameSameRerun twice. If it persists, check the prompt, the route and your own deploys.
DownSameLonger, more truncationLikely a parameter or reasoning-effort change. Check the dropped-params header and max_tokens.
SameNewChangedQuality held on this set. Widen the set before trusting it, and re-estimate cost.
Down only on fallback samplesDifferent on thoseAnyYour fallback is the weak link. See fallback equivalence.

A caveat on the served-model column: a gateway can only see a new id when the provider's reply reports one. A provider that echoes the requested name while serving something else leaves that signal blank, and only the derived metrics and your score remain 1. This is the reason the golden set exists.

Ship canary results as telemetry, not just as log lines. INFRO can push request-path spans to your own collector, and its docs list exports for the request log and usage series 6. OpenTelemetry's GenAI semantic conventions define spans, metrics and events for these calls, but they have moved to their own repository, so read attribute names from there rather than from memory 7. Emit the run score and stratum scores as metrics, and route alerts to the on-call channel your team already uses.

Decide on rollback before the first alert

The worst time to argue about thresholds is during the alert. Write the rule down, and review it as you would any other control; NIST's AI RMF is voluntary guidance, and its Generative AI Profile (NIST-AI-600-1, released July 26, 2024) proposes risk-management actions for generative AI that an organization can map its own monitoring to 8.

A rule that works in practice has three tiers:

  1. Watch. One run outside the band. Rerun immediately, double the sample count, and do nothing else. Most alerts end here.
  2. Confirm. Two consecutive runs outside the band, or one run outside plus a new served model id. Open an incident, diff failing items against the baseline, and run your larger evaluation on the old and new behavior side by side. Replaying production traffic is one way to get that sample; see replaying production traffic safely.
  3. Roll back. Confirmed drop in a critical stratum (format, safety boundary, a past-incident item), or an overall drop beyond your product's tolerance, and the larger evaluation agrees.

Rolling back is only possible if the old behavior is still reachable, and after a vendor reroute it often is not. Your options, in order: point the route at another platform that still serves the old id (the same model can have different lifecycle dates on different platforms, as Google's Gemini API and Vertex AI did for 3.7 Flash 1); move to a different pinned snapshot you have already evaluated; or keep the new model and adapt the prompt, then re-baseline. With a route, the move is a configuration change rather than a deploy; INFRO does not answer with a different model on your behalf, so the decision stays with you 9.

After any rollback or deliberate model change, freeze the old baseline, start a new one, and keep both scores in the incident record.

What the gateway can and cannot do here

INFRO is a BYOK control plane: your providers bill you directly, and the canary runs on your own connections. Three shipped parts apply here. Traffic changes raises findings for a new served snapshot, longer answers, truncation, more fallbacks and slower responses, each with a likely cause 2; it does not judge answer quality. Eval monitors score a share of served chat answers against your rubric in the background and summarize per served model, which complements the scheduled canary with a sample of live traffic. And findings can be delivered to email, Slack, signed webhooks, Teams or PagerDuty. The golden set, thresholds and rollback rule remain yours to own.

Questions readers ask

How often should canaries run?
Match the cadence to how fast you need to know and what a run costs. Daily is a sensible start for most features, hourly for revenue-critical paths. Provider notices such as a deprecation or a new snapshot are a reason to trigger an extra run on the same day, outside the schedule.
Does temperature 0 make canary results repeatable?
Not reliably. Some models refuse sampling parameters and the gateway drops them, and seed is best-effort on providers that support it 4. Treat every score as a sample from a distribution, repeat each item, and measure the noise band before setting thresholds.
Can a gateway tell me the answers got worse?
A gateway that reads only request metadata cannot; it can report that the served model, length, latency or fallback share changed 2. Quality needs an evaluation that reads answers: your golden set, or a background scorer against your rubric.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Text — OpenAI-compatible

    Available with a plan

    POST /v1/chat/completions with the request shape the OpenAI SDKs already send, including streaming and tool calling.

    Read the documentation for Text — OpenAI-compatible
  • Health-aware routing

    Available with a plan

    Named routes use priority or weighted targets over your own connections, with health checks, retries and failover.

    Read the documentation for Health-aware routing
  • Traffic changes

    Available with a plan

    Changes in your traffic that come with no error: a model answered by a new snapshot, longer answers, a lost prompt cache, more fallbacks, answers cut off by the length limit, slower responses or a new error mix. Each finding names the likely cause and the estimated weekly change in provider cost. INFRO sees request metadata, not answers, so it does not judge answer quality.

    Read the documentation for Traffic changes
  • Eval monitors

    Available with a plan

    Score a share of served chat answers against your rubric in the background with Jev or any other decision model you connect, recorded per request and summarized per served model.

    Read the documentation for Eval monitors
  • Request traces and payload logs

    Available with a plan

    Per-request timing detail, with prompt and completion logging off by default and a retention window you choose.

    Read the documentation for Request traces and payload logs
  • Usage and cost per request

    Available with a plan

    Every request records its model, latency, token or unit counts, and estimated provider cost, queryable by project and key.

    Read the documentation for Usage and cost per request
  • OpenTelemetry export

    Available with a plan

    Spans pushed to your own collector from the request path, with the endpoint and headers set in the console. Included on every plan.

    Read the documentation for OpenTelemetry export
  • Response cache

    Available with a plan

    Opt-in, exact-match cache for chat completions. An identical request inside the time-to-live you set is answered from the cache and never reaches your provider, so your provider does not bill it. Every plan; the longest time-to-live depends on the plan. No semantic matching.

    Read the documentation for Response cache

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Model aliases vs pinned snapshots: avoid silent model swaps (opens in a new tab) — INFRO. Accessed Nov 14, 2026.
  2. 2.Traffic changes — Observability (opens in a new tab) — INFRO Docs. Accessed Nov 14, 2026.
  3. 3.Working with evals (opens in a new tab) — OpenAI. Accessed Nov 14, 2026.
  4. 4.Chat completions — Modalities (opens in a new tab) — INFRO Docs. Accessed Nov 14, 2026.
  5. 5.Request traces — Observability (opens in a new tab) — INFRO Docs. Accessed Nov 14, 2026.
  6. 6.INFRO documentation overview (opens in a new tab) — INFRO Docs. Accessed Nov 14, 2026.
  7. 7.Generative AI semantic conventions (moved) (opens in a new tab) — OpenTelemetry. Accessed Nov 14, 2026.
  8. 8.AI Risk Management Framework (opens in a new tab) — NIST. Accessed Nov 14, 2026.
  9. 9.Manage AI model retirements and silent changes (opens in a new tab) — INFRO. Accessed Nov 14, 2026.

About this article

Read on 2026-11-14: INFRO's docs overview, traffic changes, request traces, chat completions and model-change use-case pages, INFRO's alias-versus-snapshot post, OpenAI's evals guide, NIST's AI RMF page and OpenTelemetry's GenAI conventions page. OpenTelemetry attribute names and vendor-side serving behavior were left out because the pages read did not confirm them; thresholds not attributed to a source are starting points, not measured values.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.