Skip to content

Evaluation and modelsMethodology

GPT-6.2 Astra and Sol: what to re-test before leaving 6.1

GPT-6.2 changes where instructions sit, what happens to reasoning after an edit, and how the model writes by default. This plan shows what to re-run on your own golden set before you move GPT-6.1 traffic, and how to judge the result in cost per successful task.

By INFRO Editorial

Published

9 min read

A single amber path on a dark ground forking into two glass branches of slightly different brightness

In short

Before moving GPT-6.1 traffic to GPT-6.2 Astra or Sol, replay a golden set through both and compare cost per successful task. Watch four things: instruction placement, reasoning behaviour, output length and token counts, and cache behaviour. OpenAI's migration notes document changes in each area 1.

Key takeaways

  • Launch benchmarks do not use your prompts, so replay a golden set built from your own traffic through GPT-6.1 and GPT-6.2 side by side.
  • GPT-6.2 moves top-level Responses instructions after input, and drops reasoning items after prefix edits; both can change caching, quality and token use.
  • At list rates GPT-6.2 Sol costs the same per token as GPT-6.1 Sol, so any cost difference there comes from output length, reasoning tokens, cache hits and retries.
  • Judge each variant on cost per successful task, not on cost per request or on a mean quality score alone.
  • Gate the switch on a canary and a rollback path, not on one clean offline run.

Why a launch post is not a regression test

A launch post tells you how a model does on someone else's tasks. Your production prompt has its own system text, tool schemas, output formats and failure modes, and a model that wins an average benchmark can still lose on your three most expensive request types. The point of a regression plan is to turn "newer and better" into a number you can defend: success rate and cost on your own inputs.

This plan covers the changes OpenAI documents in its GPT-6.2 migration guide for Astra and Sol1. One gap to state up front: the pages read for this article do not say whether the tokenizer changed between 6.1 and 6.2. Treat token counts as something to measure on your own inputs, not assume.

The plan has five steps:

  1. Build a golden set from real traffic.
  2. Run the four checks below against both models.
  3. Compute cost per successful task for each variant.
  4. Canary a slice of live traffic.
  5. Keep a rollback path.

Start with the golden set, because every later step depends on it.

Build the golden set from real requests

A golden set is a fixed list of inputs, each with an expected output or a rubric, that you replay against every candidate. Pull it from logged production requests so it carries the real distribution: typical cases, long inputs, tool-calling turns, refusals, and the odd requests that previously caused incidents.

A workable shape:

  • 150 to 300 items for a single feature, stratified by request type rather than sampled uniformly.
  • Every item tagged with metadata such as feature, language and input-size band, so you can read results per slice.
  • A deliberate block of hard cases: the ones that failed on 6.1 or needed a prompt patch.
  • Expected outputs only where a correct answer exists; otherwise a rubric.

INFRO datasets take inputs in the OpenAI chat shape with an optional expected output and metadata. You can build them from logged requests, upload JSONL or CSV, or send them from code2. Only requests whose prompt was logged can be added, so prompt logging has to have been on for the traffic you want2. Two JSONL items look like this:

JSON
{"input": {"messages": [{"role": "user", "content": "Where is my order?"}]}, "expected": "It ships tomorrow.", "metadata": {"feature": "support", "size": "short"}}
{"input": "Can I change the delivery address?", "expected": null, "metadata": {"feature": "support", "size": "short"}}

If you use the logged 6.1 answer as the expected output, you are grading 6.2 on agreement with 6.1, not on correctness. That is acceptable for spotting unintended change and wrong for deciding which model is better. Keep human-reviewed expectations for the hard slice.

Check 1: instruction placement and prefix edits

This is the change most likely to break something quietly. Per OpenAI, models starting with GPT-6.2 append top-level instructions after input in the Responses API; to make reusable instructions eligible for prompt caching, put them in a developer message as the first item in input, and keep top-level instructions for guidance that applies only to the current request1.

What to test:

  • Move stable system text into a first-position developer message and run the golden set with and without the move. Compare quality and the cached share of prompt tokens.
  • Find every place your code or your prompt manager builds instructions per request. Dynamic text there is now behind the input, so check whether it still steers the answer as it did on 6.1.
  • For multi-turn flows, replay whole conversations, not single turns.

The second change is about reasoning. Starting with GPT-6.2, the Responses API drops reasoning items after mid-conversation edits to history (prefix edits), certain model changes including downgrades, or changes to the end user or organization. OpenAI warns that answer quality can decline and the model may use more tokens to repeat earlier work1. Any code that rewrites an earlier developer message, trims history, or swaps models mid-conversation is a candidate. Our prefix-edit audit goes through the layout checks in detail. In the golden set, include multi-turn items where your application edits history, and compare token use on those turns against the same turns without the edit.

If your prompts live in a managed store, version the change. INFRO prompt management applies a versioned prompt on chat completions, Messages and Responses requests and records the version on every request, which lets you tell a prompt change from a model change when a score moves.

Check 2: reasoning behaviour and parameters

Reasoning settings do not map one-to-one. OpenAI's guidance: start with medium for a new integration, and if you currently use none or minimal, use low instead1. Run the golden set at the effort you use on 6.1 (translated under that rule) and at one level above and below. Record success rate, output tokens and latency at each level. The cheapest level that holds your success threshold is the setting to ship.

Parameter handling is a likely source of hard failures, so test it first and fix it before you compare quality:

  • For Chat Completions, omit temperature, top_p, logprobs and top_logprobs. For Responses, omit temperature, top_p and top_logprobs, and remove message.output_text.logprobs from include1.
  • Both models support Chat Completions, but tool calling requires the Responses API1. If your golden set contains tool-calling turns, confirm which endpoint your gateway or SDK sends them to before you read any result.
  • If you run experiments with a temperature or top_p on the variants, leave them out of the 6.2 variants.

Also check behaviour, not just parameters. OpenAI says Astra's writing is more concise and direct by default, and advises revisiting instructions like "be concise"1. It also says that when intent is unclear the model may propose a next step and wait for confirmation1. For an agent or batch job with no human to answer, that is a failure mode: the run ends early with a question. Add ambiguous-request items to the set and count turns that end on a proposal rather than a result.

Check 3: output length, tokenizer and token counts

Cost moves with tokens, and tokens move for three separate reasons: the tokenizer, the answer length, and reasoning tokens. Separate them, or you will not know which to fix.

  1. Input tokens for identical text. Send the same 200 inputs to both models and compare the provider-reported prompt token counts. A systematic difference is a tokenizer or template effect. The general method is in switching models changes the tokenizer.
  2. Output tokens split into visible and reasoning. Shorter default prose, which OpenAI describes for Astra1, may lower visible output while reasoning tokens go the other way. Read both from the usage block.
  3. Truncation. Count finish_reason: length per variant. A model that thinks longer can hit max_tokens on requests that fit before.

Compare distributions, not means. Report p50, p95 and the worst decile of output tokens, per request type. A mean hides the long tail that sets your bill and your timeouts.

After rollout, the same measures are worth watching live. INFRO's traffic changes read request metadata and fire on signals such as output tokens per request up by at least 15%, cached share of prompt tokens down by at least 10 points, and a rise in answers cut off by the length limit3. They read metadata only, never answers, so they tell you something changed and what it costs; they cannot tell you answers got worse3. Quality still needs your evaluators.

Check 4: cache behaviour and billing shape

The cache rules changed alongside the models. OpenAI says that migrating from GPT-5.5 or earlier means replacing prompt_cache_retention with prompt_cache_options.ttl set to "30m", and tells you to review the cache boundary and cache-write billing changes1. The pricing page lists separate short-context 30-minute and 12-hour cache-write rates for these models4. A request pattern that was a pure read on 6.1 can now include writes, so cached-share and cost both need to be measured, not inferred from the rate card.

Other billing-shape differences to check against the pages above:

  • Flex is not available on GPT-6.2 Astra or Sol1. The pricing page lists Flex rates for gpt-6.1-sol but none for the 6.2 models4.
  • Background mode with a two-hour or eight-hour window is the alternative; its rates are 30% and 40% below Standard for the 6.2 models4.
  • Data residency endpoints carry a 10% uplift for models released on or after March 5, 20264.

Enter your negotiated rates, if you have any, before comparing estimates, so the estimate reflects what your provider actually bills rather than list rates.

Compare on cost per successful task

At Standard list rates, the per-token prices tell you less than you might expect. GPT-6.2 Sol is listed at $2.00 input, $10.00 output and $0.10 cached input per 1M short-context tokens, identical to gpt-6.1-sol. GPT-6.2 Astra is listed at $8.00, $40.00 and $0.40, against $10.00, $50.00 and $1.00 for gpt-6-astra4. Those are the provider's list rates; yours may differ by contract.

For Sol, then, any cost change comes from token counts, cache hits and retries, not from the rate card. Define the metric and compute it per slice:

Text
cost per successful task = (total cost of all attempts, including retries and judge-rejected runs) / (tasks that passed your check)

The example shows why a single headline number misleads. See cost per successful task for how to count retries and human escalations as part of the cost.

Run the comparison as an experiment

INFRO experiments run a dataset through up to six variants (a model or route, a prompt version, settings) and judge every answer with up to five evaluators5. With dry_run: true you get the estimate first: the request count (items times variants, plus one judge call per evaluator per answer), the estimated provider cost per variant, and the share of your monthly request allowance it uses5. To start it you send the same body with a spend_cap of at least the estimate5. Each call is a request through the gateway on your own providers, subject to your allowlists, guardrails and spend limits5.

experiment-dry-run.json
{
  "dataset": "support-golden",
  "variants": [
    {"label": "sol-6-1", "model": "openai/gpt-6.1-sol", "prompt": {"slug": "support-reply", "version": 3}},
    {"label": "sol-6-2", "model": "openai/gpt-6.2-sol", "prompt": {"slug": "support-reply", "version": 4}},
    {"label": "astra-6-2", "model": "openai/gpt-6.2-astra", "prompt": {"slug": "support-reply", "version": 4}}
  ],
  "evaluators": ["helpfulness"],
  "dry_run": true
}

Use a separate prompt version for 6.2 only if you moved instructions or loosened a "be concise" line, so the model and prompt effects stay distinguishable. If prompt version 4 is the only change, add a fourth variant of 6.1 with version 4 to isolate it. Judges are models too; check one against human labels on a sample before trusting it, as described in calibrating graders against humans.

Canary and keep a way back

An offline pass is a gate, not a verdict. Send a small share of live traffic to the 6.2 model, split by a stable key so a user stays on one variant, and watch success proxies (thumbs-down share, retries, escalations), output tokens, cached share and latency for several days. Replaying production traffic covers what is safe to replay.

Set the stop conditions in advance, in writing:

  • Success rate on the hard slice falls below your threshold.
  • p95 output tokens rise past the level your timeout or max_tokens can absorb.
  • Cost per successful task exceeds the 6.1 baseline by more than you agreed.

Pin the exact model id on both sides and keep the 6.1 route intact until the canary closes; aliases versus pinned snapshots explains why. INFRO scores and prompt rollback rules can move a prompt label back to the previous version when its scores fall below a threshold, which covers prompt regressions but does not by itself change which model a route calls. Keep the model rollback as a separate, tested step.

Questions readers ask

Does GPT-6.2 Sol cost the same as GPT-6.1 Sol?
Per token at Standard list rates, yes: OpenAI's pricing page lists both at $2.00 input and $10.00 output per 1M short-context tokens 4. What you pay per task can still differ, because output length, reasoning tokens, cache hits and retries differ between models.
Can I keep using Flex processing after the move?
Not on these models. OpenAI states that GPT-6.2 Astra and Sol do not support Flex processing, and offers background mode with two-hour or eight-hour completion windows as the alternative 1. If your 6.1 traffic uses Flex, price that change separately.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Datasets

    Available with a plan

    Inputs and expected outputs built from logged requests in one click, uploaded as JSONL or CSV, or sent from code, kept until you delete them. On every plan.

  • Experiments

    Available with a plan

    A dataset run through the gateway across up to six variants (a model or route, a prompt version, settings), judged by your evaluators and compared side by side, with the request count and estimated provider cost shown before it starts and a spend cap it stops at. On every plan.

  • Evaluators

    Available with a plan

    A rubric a chat model judges answers by, on your own provider account: a sample of live traffic held to a monthly spend limit, or up to 1,000 logged requests on demand with an estimate first and a spend cap. Each verdict is a score on the request it judged, and each judge call is a request on your provider, counted against your allowance. On every plan.

  • Usage and cost per request

    Available with a plan

    Every request records its model, latency, token or unit counts, and estimated provider cost, queryable by project and key.

    Read the documentation for Usage and cost per request
  • Traffic changes

    Available with a plan

    Changes in your traffic that come with no error: a model answered by a new snapshot, longer answers, a lost prompt cache, more fallbacks, answers cut off by the length limit, slower responses or a new error mix. Each finding names the likely cause and the estimated weekly change in provider cost. INFRO sees request metadata, not answers, so it does not judge answer quality.

    Read the documentation for Traffic changes
  • Prompt management

    Available with a plan

    Versioned prompts with labels such as production, filled and applied by the gateway on chat completions, Messages and Responses requests before routing, guardrails and spend checks, with the version recorded on every request. A protected label is moved only by a person in the console. On every plan.

  • Negotiated provider rates

    Available with a plan

    Enter the rates your contracts actually carry, per model and optionally per provider, so estimated cost uses them before the vendor's list rate. Included on every plan.

    Read the documentation for Negotiated provider rates

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Using GPT-6.2 (opens in a new tab) — OpenAI. Accessed Nov 19, 2026.
  2. 2.Datasets (opens in a new tab) — INFRO. Accessed Nov 19, 2026.
  3. 3.Traffic changes (opens in a new tab) — INFRO. Accessed Nov 19, 2026.
  4. 4.Pricing (opens in a new tab) — OpenAI. Accessed Nov 19, 2026.
  5. 5.Experiments (opens in a new tab) — INFRO. Accessed Nov 19, 2026.

About this article

Read on 2026-11-19: OpenAI's "Using GPT-6.2" migration guide and API pricing page (markdown versions), plus INFRO's docs on datasets, experiments and traffic changes. OpenAI's launch announcement returned HTTP 403, so no launch benchmark or claim from it is used; tokenizer changes are not stated on the pages read and are treated as something to measure.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.