Why a pinned id still drifts
Pinning protects you from upgrades the vendor announces by name. It does not protect you from three other things. First, a stable id can be rerouted: on 8 October 2026 Google began answering requests for gemini-3.7-flash with gemini-3.8-flash, and xAI sends grok-voice-transcribe-1.0 requests to 2.0 since its end of life 1. Second, your own routing can change what answers: a fallback target, a rule that sends some requests elsewhere, an auto-routed alias. Third, the same model id can report differently on different providers, so a failover changes the serving path without changing the name you sent.
None of these raises an error. INFRO's traffic-changes docs describe the same class of problem: a model name moves to a new snapshot and answers get longer, or a primary times out for a few requests and they quietly move to a fallback 2. Metadata catches the shape of that. It cannot tell you the answers got worse, because the gateway reads request metadata, not prompts or answers 2. For that you need a fixed set of questions with known-good outcomes, asked repeatedly, through the same path production uses.
This post covers that second half: the golden set, the canary schedule, the thresholds, and the rollback rule. For the naming conventions and pinning rules, see model aliases vs pinned snapshots.
Build a golden set that fits in one afternoon
A golden set is a small, frozen list of inputs with a machine-checkable definition of "good". Start with 40 to 100 items. Fewer than 40 and one flaky item moves the score by a visible amount; more than 100 and you will stop maintaining it.
OpenAI's evals guide describes the same minimal shape: a schema for each test item and testing criteria such as a string check against a human-supplied label 3. That platform is being retired: it became read-only on October 31, 2026 and is scheduled to shut down on November 30, 2026 3. Keep your golden set in your own repository as plain JSONL so it outlives any vendor's eval tool; see export your evals and rebuild them in CI.
Draw items from four places:
- Real production prompts, scrubbed of personal data, stratified by feature and by input length.
- Format-critical prompts: anything whose output a parser consumes (JSON, enums, tool-call arguments).
- Boundary prompts: requests that sit near a refusal line or a policy rule, where a changed safety profile shows first.
- Past incidents: every prompt that once produced a bad answer in production, with the fix as the expected outcome.
Prefer deterministic checks (exact label, JSON that parses and has the required keys, a regex, a length bound, a required citation) for at least 70% of items. Use a model judge only for the remainder, and calibrate it first; calibrating graders against humans covers that.
Version the file. Every change to the set resets the baseline, so record the set's hash next to each result and never compare scores across two hashes.
Run canaries through the gateway, not around it
A canary that calls the provider directly measures the provider. A canary that goes through your gateway, on the route production uses, measures what your users get, including routing rules, fallbacks and parameter handling. The gateway's chat endpoint takes either a route slug or a vendor/model-name id in the model field 4, so the canary can call the exact route name production calls.
Four details make the result trustworthy:
- Dedicated key and project. Give the canary its own gateway key in the same environment as production so its cost and requests are separable in usage and the request log. The request log records the model, the connection that served it, outcome, timings, usage and estimated cost, and can be queried per key 5.
- No response cache on the canary path. An exact-match cache answers an identical request without reaching the provider, and cache hits are excluded from traffic-change signals for that reason 2. A cached canary would score your cache. Keep caching off for the canary key or route.
- Sample each item several times. Sampling parameters are not always honored: some current models refuse temperature, top_p and top_k, and the gateway drops them and lists what it dropped in the
X-INFRO-Params-Droppedheader;seedis best-effort 4. Run each item three to five times and store the per-item pass rate. - Record what answered. Store the model id in the response body, the finish reason, completion tokens, latency and the dropped-params header with every sample. The model id is the first evidence of a swap 1.
import json, time, re
from openai import OpenAI
client = OpenAI(base_url="https://api.infro.io/v1", api_key=CANARY_KEY)
ROUTE = "support-triage" # the same route slug production calls
K = 3
def check(item, text):
kind = item["check"]
if kind == "exact":
return text.strip() == item["expected"]
if kind == "regex":
return re.search(item["expected"], text) is not None
if kind == "json_keys":
try:
obj = json.loads(text)
except ValueError:
return False
return all(k in obj for k in item["expected"])
return False
def run_item(item):
samples = []
for _ in range(K):
t0 = time.time()
raw = client.chat.completions.with_raw_response.create(
model=ROUTE,
messages=item["messages"],
max_tokens=item.get("max_tokens", 400),
temperature=0,
)
resp = raw.parse()
text = resp.choices[0].message.content or ""
samples.append({
"ok": check(item, text),
"served_model": resp.model,
"finish": resp.choices[0].finish_reason,
"out_tokens": resp.usage.completion_tokens,
"latency_s": round(time.time() - t0, 3),
"dropped": raw.headers.get("X-INFRO-Params-Dropped"),
})
return {"id": item["id"], "stratum": item["stratum"], "samples": samples}Schedule it from cron or your CI scheduler, append one JSON line per run to storage you control, and compute the run score as the mean of per-item pass rates. Cap the canary key with a budget so a looping scheduler cannot spend without limit.
Set thresholds from your own noise
A fixed rule such as "alert below 90%" fails twice: it fires constantly on a noisy set and never on a stable set that slides from 99% to 91%. Measure the noise first.
Run the full canary for at least 14 scheduled runs with nothing changed. Take the mean and standard deviation of the run score, per stratum as well as overall. Then alert only when the drop is both statistically large and practically large:
import statistics
def drifted(history, today, min_drop=0.05):
base = history[-14:]
mu = statistics.mean(base)
sd = max(statistics.stdev(base), 0.01)
return today < mu - max(3 * sd, min_drop)The min_drop floor stops a very quiet baseline from paging on a one-item wobble. Apply the rule per stratum: a six-point drop confined to the structured-output stratum is a real signal that an overall mean would hide. For format-critical strata, add a hard floor instead of a band: if a parser consumes the output, a pass rate below your own tolerance is an incident whatever the history says.
Pair the score with the metadata signals that need no judge. INFRO's traffic-changes docs give workable reference points: output tokens per request up at least 15% and 20 tokens, finish_reason: length share up 3 percentage points, fallback share up 5 points, p95 latency up 25% and 300 ms two days running 2. Those are the gateway's own thresholds; take them as a starting point and tune against your canary's variance.
| Signal | Source | Starting threshold |
|---|---|---|
| Run score, per stratum | Canary | Below baseline mean minus the larger of 3 SD or 5 points |
| Served model id | Canary and request log | Any id not seen for this route in the last 14 days |
| Output tokens per sample | Canary | Up 15% over baseline median |
| Truncated samples | Canary | Up 3 points |
| Parse failures | Canary | Any rise above your tolerance |
Read the signals together
Each signal is ambiguous alone, and the combination usually names the cause.
| Score | Served model id | Length or tokens | What to do first |
|---|---|---|---|
| Down | New | Changed | Treat as a snapshot swap. Run the full evaluation on the new id before deciding. |
| Down | Same | Same | Rerun twice. If it persists, check the prompt, the route and your own deploys. |
| Down | Same | Longer, more truncation | Likely a parameter or reasoning-effort change. Check the dropped-params header and max_tokens. |
| Same | New | Changed | Quality held on this set. Widen the set before trusting it, and re-estimate cost. |
| Down only on fallback samples | Different on those | Any | Your fallback is the weak link. See fallback equivalence. |
A caveat on the served-model column: a gateway can only see a new id when the provider's reply reports one. A provider that echoes the requested name while serving something else leaves that signal blank, and only the derived metrics and your score remain 1. This is the reason the golden set exists.
Ship canary results as telemetry, not just as log lines. INFRO can push request-path spans to your own collector, and its docs list exports for the request log and usage series 6. OpenTelemetry's GenAI semantic conventions define spans, metrics and events for these calls, but they have moved to their own repository, so read attribute names from there rather than from memory 7. Emit the run score and stratum scores as metrics, and route alerts to the on-call channel your team already uses.
Decide on rollback before the first alert
The worst time to argue about thresholds is during the alert. Write the rule down, and review it as you would any other control; NIST's AI RMF is voluntary guidance, and its Generative AI Profile (NIST-AI-600-1, released July 26, 2024) proposes risk-management actions for generative AI that an organization can map its own monitoring to 8.
A rule that works in practice has three tiers:
- Watch. One run outside the band. Rerun immediately, double the sample count, and do nothing else. Most alerts end here.
- Confirm. Two consecutive runs outside the band, or one run outside plus a new served model id. Open an incident, diff failing items against the baseline, and run your larger evaluation on the old and new behavior side by side. Replaying production traffic is one way to get that sample; see replaying production traffic safely.
- Roll back. Confirmed drop in a critical stratum (format, safety boundary, a past-incident item), or an overall drop beyond your product's tolerance, and the larger evaluation agrees.
Rolling back is only possible if the old behavior is still reachable, and after a vendor reroute it often is not. Your options, in order: point the route at another platform that still serves the old id (the same model can have different lifecycle dates on different platforms, as Google's Gemini API and Vertex AI did for 3.7 Flash 1); move to a different pinned snapshot you have already evaluated; or keep the new model and adapt the prompt, then re-baseline. With a route, the move is a configuration change rather than a deploy; INFRO does not answer with a different model on your behalf, so the decision stays with you 9.
After any rollback or deliberate model change, freeze the old baseline, start a new one, and keep both scores in the incident record.
What the gateway can and cannot do here
INFRO is a BYOK control plane: your providers bill you directly, and the canary runs on your own connections. Three shipped parts apply here. Traffic changes raises findings for a new served snapshot, longer answers, truncation, more fallbacks and slower responses, each with a likely cause 2; it does not judge answer quality. Eval monitors score a share of served chat answers against your rubric in the background and summarize per served model, which complements the scheduled canary with a sample of live traffic. And findings can be delivered to email, Slack, signed webhooks, Teams or PagerDuty. The golden set, thresholds and rollback rule remain yours to own.
