Why a launch post is not a regression test
A launch post tells you how a model does on someone else's tasks. Your production prompt has its own system text, tool schemas, output formats and failure modes, and a model that wins an average benchmark can still lose on your three most expensive request types. The point of a regression plan is to turn "newer and better" into a number you can defend: success rate and cost on your own inputs.
This plan covers the changes OpenAI documents in its GPT-6.2 migration guide for Astra and Sol1. One gap to state up front: the pages read for this article do not say whether the tokenizer changed between 6.1 and 6.2. Treat token counts as something to measure on your own inputs, not assume.
The plan has five steps:
- Build a golden set from real traffic.
- Run the four checks below against both models.
- Compute cost per successful task for each variant.
- Canary a slice of live traffic.
- Keep a rollback path.
Start with the golden set, because every later step depends on it.
Build the golden set from real requests
A golden set is a fixed list of inputs, each with an expected output or a rubric, that you replay against every candidate. Pull it from logged production requests so it carries the real distribution: typical cases, long inputs, tool-calling turns, refusals, and the odd requests that previously caused incidents.
A workable shape:
- 150 to 300 items for a single feature, stratified by request type rather than sampled uniformly.
- Every item tagged with metadata such as feature, language and input-size band, so you can read results per slice.
- A deliberate block of hard cases: the ones that failed on 6.1 or needed a prompt patch.
- Expected outputs only where a correct answer exists; otherwise a rubric.
INFRO datasets take inputs in the OpenAI chat shape with an optional expected output and metadata. You can build them from logged requests, upload JSONL or CSV, or send them from code2. Only requests whose prompt was logged can be added, so prompt logging has to have been on for the traffic you want2. Two JSONL items look like this:
{"input": {"messages": [{"role": "user", "content": "Where is my order?"}]}, "expected": "It ships tomorrow.", "metadata": {"feature": "support", "size": "short"}}
{"input": "Can I change the delivery address?", "expected": null, "metadata": {"feature": "support", "size": "short"}}If you use the logged 6.1 answer as the expected output, you are grading 6.2 on agreement with 6.1, not on correctness. That is acceptable for spotting unintended change and wrong for deciding which model is better. Keep human-reviewed expectations for the hard slice.
Check 1: instruction placement and prefix edits
This is the change most likely to break something quietly. Per OpenAI, models starting with GPT-6.2 append top-level instructions after input in the Responses API; to make reusable instructions eligible for prompt caching, put them in a developer message as the first item in input, and keep top-level instructions for guidance that applies only to the current request1.
What to test:
- Move stable system text into a first-position
developermessage and run the golden set with and without the move. Compare quality and the cached share of prompt tokens. - Find every place your code or your prompt manager builds
instructionsper request. Dynamic text there is now behind the input, so check whether it still steers the answer as it did on 6.1. - For multi-turn flows, replay whole conversations, not single turns.
The second change is about reasoning. Starting with GPT-6.2, the Responses API drops reasoning items after mid-conversation edits to history (prefix edits), certain model changes including downgrades, or changes to the end user or organization. OpenAI warns that answer quality can decline and the model may use more tokens to repeat earlier work1. Any code that rewrites an earlier developer message, trims history, or swaps models mid-conversation is a candidate. Our prefix-edit audit goes through the layout checks in detail. In the golden set, include multi-turn items where your application edits history, and compare token use on those turns against the same turns without the edit.
If your prompts live in a managed store, version the change. INFRO prompt management applies a versioned prompt on chat completions, Messages and Responses requests and records the version on every request, which lets you tell a prompt change from a model change when a score moves.
Check 2: reasoning behaviour and parameters
Reasoning settings do not map one-to-one. OpenAI's guidance: start with medium for a new integration, and if you currently use none or minimal, use low instead1. Run the golden set at the effort you use on 6.1 (translated under that rule) and at one level above and below. Record success rate, output tokens and latency at each level. The cheapest level that holds your success threshold is the setting to ship.
Parameter handling is a likely source of hard failures, so test it first and fix it before you compare quality:
- For Chat Completions, omit
temperature,top_p,logprobsandtop_logprobs. For Responses, omittemperature,top_pandtop_logprobs, and removemessage.output_text.logprobsfrominclude1. - Both models support Chat Completions, but tool calling requires the Responses API1. If your golden set contains tool-calling turns, confirm which endpoint your gateway or SDK sends them to before you read any result.
- If you run experiments with a
temperatureortop_pon the variants, leave them out of the 6.2 variants.
Also check behaviour, not just parameters. OpenAI says Astra's writing is more concise and direct by default, and advises revisiting instructions like "be concise"1. It also says that when intent is unclear the model may propose a next step and wait for confirmation1. For an agent or batch job with no human to answer, that is a failure mode: the run ends early with a question. Add ambiguous-request items to the set and count turns that end on a proposal rather than a result.
Check 3: output length, tokenizer and token counts
Cost moves with tokens, and tokens move for three separate reasons: the tokenizer, the answer length, and reasoning tokens. Separate them, or you will not know which to fix.
- Input tokens for identical text. Send the same 200 inputs to both models and compare the provider-reported prompt token counts. A systematic difference is a tokenizer or template effect. The general method is in switching models changes the tokenizer.
- Output tokens split into visible and reasoning. Shorter default prose, which OpenAI describes for Astra1, may lower visible output while reasoning tokens go the other way. Read both from the usage block.
- Truncation. Count
finish_reason: lengthper variant. A model that thinks longer can hitmax_tokenson requests that fit before.
Compare distributions, not means. Report p50, p95 and the worst decile of output tokens, per request type. A mean hides the long tail that sets your bill and your timeouts.
After rollout, the same measures are worth watching live. INFRO's traffic changes read request metadata and fire on signals such as output tokens per request up by at least 15%, cached share of prompt tokens down by at least 10 points, and a rise in answers cut off by the length limit3. They read metadata only, never answers, so they tell you something changed and what it costs; they cannot tell you answers got worse3. Quality still needs your evaluators.
Check 4: cache behaviour and billing shape
The cache rules changed alongside the models. OpenAI says that migrating from GPT-5.5 or earlier means replacing prompt_cache_retention with prompt_cache_options.ttl set to "30m", and tells you to review the cache boundary and cache-write billing changes1. The pricing page lists separate short-context 30-minute and 12-hour cache-write rates for these models4. A request pattern that was a pure read on 6.1 can now include writes, so cached-share and cost both need to be measured, not inferred from the rate card.
Other billing-shape differences to check against the pages above:
- Flex is not available on GPT-6.2 Astra or Sol1. The pricing page lists Flex rates for gpt-6.1-sol but none for the 6.2 models4.
- Background mode with a two-hour or eight-hour window is the alternative; its rates are 30% and 40% below Standard for the 6.2 models4.
- Data residency endpoints carry a 10% uplift for models released on or after March 5, 20264.
Enter your negotiated rates, if you have any, before comparing estimates, so the estimate reflects what your provider actually bills rather than list rates.
Compare on cost per successful task
At Standard list rates, the per-token prices tell you less than you might expect. GPT-6.2 Sol is listed at $2.00 input, $10.00 output and $0.10 cached input per 1M short-context tokens, identical to gpt-6.1-sol. GPT-6.2 Astra is listed at $8.00, $40.00 and $0.40, against $10.00, $50.00 and $1.00 for gpt-6-astra4. Those are the provider's list rates; yours may differ by contract.
For Sol, then, any cost change comes from token counts, cache hits and retries, not from the rate card. Define the metric and compute it per slice:
cost per successful task = (total cost of all attempts, including retries and judge-rejected runs) / (tasks that passed your check)The example shows why a single headline number misleads. See cost per successful task for how to count retries and human escalations as part of the cost.
Run the comparison as an experiment
INFRO experiments run a dataset through up to six variants (a model or route, a prompt version, settings) and judge every answer with up to five evaluators5. With dry_run: true you get the estimate first: the request count (items times variants, plus one judge call per evaluator per answer), the estimated provider cost per variant, and the share of your monthly request allowance it uses5. To start it you send the same body with a spend_cap of at least the estimate5. Each call is a request through the gateway on your own providers, subject to your allowlists, guardrails and spend limits5.
{
"dataset": "support-golden",
"variants": [
{"label": "sol-6-1", "model": "openai/gpt-6.1-sol", "prompt": {"slug": "support-reply", "version": 3}},
{"label": "sol-6-2", "model": "openai/gpt-6.2-sol", "prompt": {"slug": "support-reply", "version": 4}},
{"label": "astra-6-2", "model": "openai/gpt-6.2-astra", "prompt": {"slug": "support-reply", "version": 4}}
],
"evaluators": ["helpfulness"],
"dry_run": true
}Use a separate prompt version for 6.2 only if you moved instructions or loosened a "be concise" line, so the model and prompt effects stay distinguishable. If prompt version 4 is the only change, add a fourth variant of 6.1 with version 4 to isolate it. Judges are models too; check one against human labels on a sample before trusting it, as described in calibrating graders against humans.
Canary and keep a way back
An offline pass is a gate, not a verdict. Send a small share of live traffic to the 6.2 model, split by a stable key so a user stays on one variant, and watch success proxies (thumbs-down share, retries, escalations), output tokens, cached share and latency for several days. Replaying production traffic covers what is safe to replay.
Set the stop conditions in advance, in writing:
- Success rate on the hard slice falls below your threshold.
- p95 output tokens rise past the level your timeout or
max_tokenscan absorb. - Cost per successful task exceeds the 6.1 baseline by more than you agreed.
Pin the exact model id on both sides and keep the 6.1 route intact until the canary closes; aliases versus pinned snapshots explains why. INFRO scores and prompt rollback rules can move a prompt label back to the previous version when its scores fall below a threshold, which covers prompt regressions but does not by itself change which model a route calls. Keep the model rollback as a separate, tested step.
