# GPT-6.2 Astra and Sol: what to re-test before leaving 6.1

URL: https://infro.io/blog/gpt-6-2-astra-and-sol-what-to-re-test-before-leaving-6-1
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-19
Last modified: 2026-11-19
Last reviewed: 2026-11-19 (next review due 2026-12-19)
Topic: Evaluation and models
Format: Methodology

> Before moving GPT-6.1 traffic to GPT-6.2 Astra or Sol, replay a golden set through both and compare cost per successful task. Watch four things: instruction placement, reasoning behaviour, output length and token counts, and cache behaviour. OpenAI's migration notes document changes in each area [^openai-gpt-62].

Key takeaways:
- Launch benchmarks do not use your prompts, so replay a golden set built from your own traffic through GPT-6.1 and GPT-6.2 side by side.
- GPT-6.2 moves top-level Responses `instructions` after `input`, and drops reasoning items after prefix edits; both can change caching, quality and token use.
- At list rates GPT-6.2 Sol costs the same per token as GPT-6.1 Sol, so any cost difference there comes from output length, reasoning tokens, cache hits and retries.
- Judge each variant on cost per successful task, not on cost per request or on a mean quality score alone.
- Gate the switch on a canary and a rollback path, not on one clean offline run.

## Why a launch post is not a regression test

A launch post tells you how a model does on someone else's tasks. Your production prompt has its own system text, tool schemas, output formats and failure modes, and a model that wins an average benchmark can still lose on your three most expensive request types. The point of a regression plan is to turn "newer and better" into a number you can defend: success rate and cost on your own inputs.

This plan covers the changes OpenAI documents in its GPT-6.2 migration guide for Astra and Sol[1]. One gap to state up front: the pages read for this article do not say whether the tokenizer changed between 6.1 and 6.2. Treat token counts as something to measure on your own inputs, not assume.

The plan has five steps:

1. Build a golden set from real traffic.
2. Run the four checks below against both models.
3. Compute cost per successful task for each variant.
4. Canary a slice of live traffic.
5. Keep a rollback path.

Start with the golden set, because every later step depends on it.

## Build the golden set from real requests

A golden set is a fixed list of inputs, each with an expected output or a rubric, that you replay against every candidate. Pull it from logged production requests so it carries the real distribution: typical cases, long inputs, tool-calling turns, refusals, and the odd requests that previously caused incidents.

A workable shape:

- 150 to 300 items for a single feature, stratified by request type rather than sampled uniformly.
- Every item tagged with metadata such as feature, language and input-size band, so you can read results per slice.
- A deliberate block of hard cases: the ones that failed on 6.1 or needed a prompt patch.
- Expected outputs only where a correct answer exists; otherwise a rubric.

INFRO datasets take inputs in the OpenAI chat shape with an optional expected output and metadata. You can build them from logged requests, upload JSONL or CSV, or send them from code[2]. Only requests whose prompt was logged can be added, so prompt logging has to have been on for the traffic you want[2]. Two JSONL items look like this:

```json
{"input": {"messages": [{"role": "user", "content": "Where is my order?"}]}, "expected": "It ships tomorrow.", "metadata": {"feature": "support", "size": "short"}}
{"input": "Can I change the delivery address?", "expected": null, "metadata": {"feature": "support", "size": "short"}}
```

If you use the logged 6.1 answer as the expected output, you are grading 6.2 on agreement with 6.1, not on correctness. That is acceptable for spotting unintended change and wrong for deciding which model is better. Keep human-reviewed expectations for the hard slice.

## Check 1: instruction placement and prefix edits

This is the change most likely to break something quietly. Per OpenAI, models starting with GPT-6.2 append top-level `instructions` after `input` in the Responses API; to make reusable instructions eligible for prompt caching, put them in a `developer` message as the first item in `input`, and keep top-level `instructions` for guidance that applies only to the current request[1].

What to test:

- Move stable system text into a first-position `developer` message and run the golden set with and without the move. Compare quality and the cached share of prompt tokens.
- Find every place your code or your prompt manager builds `instructions` per request. Dynamic text there is now behind the input, so check whether it still steers the answer as it did on 6.1.
- For multi-turn flows, replay whole conversations, not single turns.

The second change is about reasoning. Starting with GPT-6.2, the Responses API drops reasoning items after mid-conversation edits to history (prefix edits), certain model changes including downgrades, or changes to the end user or organization. OpenAI warns that answer quality can decline and the model may use more tokens to repeat earlier work[1]. Any code that rewrites an earlier developer message, trims history, or swaps models mid-conversation is a candidate. Our [prefix-edit audit](https://infro.io/blog/gpt-6-2-and-prefix-edits-audit-your-prompt-layout) goes through the layout checks in detail. In the golden set, include multi-turn items where your application edits history, and compare token use on those turns against the same turns without the edit.

If your prompts live in a managed store, version the change. INFRO prompt management applies a versioned prompt on chat completions, Messages and Responses requests and records the version on every request, which lets you tell a prompt change from a model change when a score moves.

## Check 2: reasoning behaviour and parameters

Reasoning settings do not map one-to-one. OpenAI's guidance: start with `medium` for a new integration, and if you currently use `none` or `minimal`, use `low` instead[1]. Run the golden set at the effort you use on 6.1 (translated under that rule) and at one level above and below. Record success rate, output tokens and latency at each level. The cheapest level that holds your success threshold is the setting to ship.

Parameter handling is a likely source of hard failures, so test it first and fix it before you compare quality:

- For Chat Completions, omit `temperature`, `top_p`, `logprobs` and `top_logprobs`. For Responses, omit `temperature`, `top_p` and `top_logprobs`, and remove `message.output_text.logprobs` from `include`[1].
- Both models support Chat Completions, but tool calling requires the Responses API[1]. If your golden set contains tool-calling turns, confirm which endpoint your gateway or SDK sends them to before you read any result.
- If you run experiments with a `temperature` or `top_p` on the variants, leave them out of the 6.2 variants.

Also check behaviour, not just parameters. OpenAI says Astra's writing is more concise and direct by default, and advises revisiting instructions like "be concise"[1]. It also says that when intent is unclear the model may propose a next step and wait for confirmation[1]. For an agent or batch job with no human to answer, that is a failure mode: the run ends early with a question. Add ambiguous-request items to the set and count turns that end on a proposal rather than a result.

## Check 3: output length, tokenizer and token counts

Cost moves with tokens, and tokens move for three separate reasons: the tokenizer, the answer length, and reasoning tokens. Separate them, or you will not know which to fix.

1. **Input tokens for identical text.** Send the same 200 inputs to both models and compare the provider-reported prompt token counts. A systematic difference is a tokenizer or template effect. The general method is in [switching models changes the tokenizer](https://infro.io/blog/switching-models-changes-the-tokenizer-how-much-does-your-bill-move).
2. **Output tokens split into visible and reasoning.** Shorter default prose, which OpenAI describes for Astra[1], may lower visible output while reasoning tokens go the other way. Read both from the usage block.
3. **Truncation.** Count `finish_reason: length` per variant. A model that thinks longer can hit `max_tokens` on requests that fit before.

Compare distributions, not means. Report p50, p95 and the worst decile of output tokens, per request type. A mean hides the long tail that sets your bill and your timeouts.

After rollout, the same measures are worth watching live. INFRO's traffic changes read request metadata and fire on signals such as output tokens per request up by at least 15%, cached share of prompt tokens down by at least 10 points, and a rise in answers cut off by the length limit[3]. They read metadata only, never answers, so they tell you something changed and what it costs; they cannot tell you answers got worse[3]. Quality still needs your evaluators.

## Check 4: cache behaviour and billing shape

The cache rules changed alongside the models. OpenAI says that migrating from GPT-5.5 or earlier means replacing `prompt_cache_retention` with `prompt_cache_options.ttl` set to `"30m"`, and tells you to review the cache boundary and cache-write billing changes[1]. The pricing page lists separate short-context 30-minute and 12-hour cache-write rates for these models[4]. A request pattern that was a pure read on 6.1 can now include writes, so cached-share and cost both need to be measured, not inferred from the rate card.

Other billing-shape differences to check against the pages above:

- Flex is not available on GPT-6.2 Astra or Sol[1]. The pricing page lists Flex rates for gpt-6.1-sol but none for the 6.2 models[4].
- Background mode with a two-hour or eight-hour window is the alternative; its rates are 30% and 40% below Standard for the 6.2 models[4].
- Data residency endpoints carry a 10% uplift for models released on or after March 5, 2026[4].

Enter your negotiated rates, if you have any, before comparing estimates, so the estimate reflects what your provider actually bills rather than list rates.

## Compare on cost per successful task

At Standard list rates, the per-token prices tell you less than you might expect. GPT-6.2 Sol is listed at $2.00 input, $10.00 output and $0.10 cached input per 1M short-context tokens, identical to gpt-6.1-sol. GPT-6.2 Astra is listed at $8.00, $40.00 and $0.40, against $10.00, $50.00 and $1.00 for gpt-6-astra[4]. Those are the provider's list rates; yours may differ by contract.

For Sol, then, any cost change comes from token counts, cache hits and retries, not from the rate card. Define the metric and compute it per slice:

```text
cost per successful task = (total cost of all attempts, including retries and judge-rejected runs) / (tasks that passed your check)
```

> **Example: Illustrative numbers, not measurements**
> Per task at list rates, ignoring caching: GPT-6.1 Sol with 3,000 input and 1,200 output tokens costs $0.018, and with an 82% pass rate that is about $0.0220 per successful task. A GPT-6.2 Sol run that passes 88% but writes 1,500 output tokens costs $0.021, or about $0.0239 per success. A higher pass rate did not lower cost per success because output grew. An Astra run with 900 output tokens at $8 and $40 costs $0.060, and at 94% that is about $0.0638 per success. Whether the extra pass rate is worth that depends on what a failed task costs you.

The example shows why a single headline number misleads. See [cost per successful task](https://infro.io/blog/cost-per-successful-task) for how to count retries and human escalations as part of the cost.

## Run the comparison as an experiment

INFRO experiments run a dataset through up to six variants (a model or route, a prompt version, settings) and judge every answer with up to five evaluators[5]. With `dry_run: true` you get the estimate first: the request count (items times variants, plus one judge call per evaluator per answer), the estimated provider cost per variant, and the share of your monthly request allowance it uses[5]. To start it you send the same body with a `spend_cap` of at least the estimate[5]. Each call is a request through the gateway on your own providers, subject to your allowlists, guardrails and spend limits[5].

```json
{
  "dataset": "support-golden",
  "variants": [
    {"label": "sol-6-1", "model": "openai/gpt-6.1-sol", "prompt": {"slug": "support-reply", "version": 3}},
    {"label": "sol-6-2", "model": "openai/gpt-6.2-sol", "prompt": {"slug": "support-reply", "version": 4}},
    {"label": "astra-6-2", "model": "openai/gpt-6.2-astra", "prompt": {"slug": "support-reply", "version": 4}}
  ],
  "evaluators": ["helpfulness"],
  "dry_run": true
}
```

Use a separate prompt version for 6.2 only if you moved instructions or loosened a "be concise" line, so the model and prompt effects stay distinguishable. If prompt version 4 is the only change, add a fourth variant of 6.1 with version 4 to isolate it. Judges are models too; check one against human labels on a sample before trusting it, as described in [calibrating graders against humans](https://infro.io/blog/can-an-llm-judge-be-trusted-calibrating-graders-against-humans).

## Canary and keep a way back

An offline pass is a gate, not a verdict. Send a small share of live traffic to the 6.2 model, split by a stable key so a user stays on one variant, and watch success proxies (thumbs-down share, retries, escalations), output tokens, cached share and latency for several days. [Replaying production traffic](https://infro.io/blog/can-you-replay-production-traffic-to-test-a-new-model-safely) covers what is safe to replay.

Set the stop conditions in advance, in writing:

- Success rate on the hard slice falls below your threshold.
- p95 output tokens rise past the level your timeout or `max_tokens` can absorb.
- Cost per successful task exceeds the 6.1 baseline by more than you agreed.

Pin the exact model id on both sides and keep the 6.1 route intact until the canary closes; [aliases versus pinned snapshots](https://infro.io/blog/model-aliases-vs-pinned-snapshots) explains why. INFRO scores and prompt rollback rules can move a prompt label back to the previous version when its scores fall below a threshold, which covers prompt regressions but does not by itself change which model a route calls. Keep the model rollback as a separate, tested step.

## Questions

### Does GPT-6.2 Sol cost the same as GPT-6.1 Sol?

Per token at Standard list rates, yes: OpenAI's pricing page lists both at $2.00 input and $10.00 output per 1M short-context tokens [4]. What you pay per task can still differ, because output length, reasoning tokens, cache hits and retries differ between models.

### Can I keep using Flex processing after the move?

Not on these models. OpenAI states that GPT-6.2 Astra and Sol do not support Flex processing, and offers background mode with two-hour or eight-hour completion windows as the alternative [1]. If your 6.1 traffic uses Flex, price that change separately.

## Sources

1. Using GPT-6.2 — OpenAI. https://developers.openai.com/api/docs/guides/latest-model.md (accessed 2026-11-19)
2. Datasets — INFRO. https://infro.io/docs/observability/datasets (accessed 2026-11-19)
3. Traffic changes — INFRO. https://infro.io/docs/observability/traffic-changes (accessed 2026-11-19)
4. Pricing — OpenAI. https://developers.openai.com/api/docs/pricing.md (accessed 2026-11-19)
5. Experiments — INFRO. https://infro.io/docs/observability/experiments (accessed 2026-11-19)
