# Testing GPT-Live-1.1: a recorded-session method for voice agents

URL: https://infro.io/blog/testing-gpt-live-1-1-a-recorded-session-method-for-voice-agents
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-11-21
Last modified: 2026-11-21
Last reviewed: 2026-11-21 (next review due 2026-12-21)
Topic: Evaluation and models
Format: Methodology

> Replay the same recorded caller audio through your current GPT-Live model and GPT-Live-1.1, repeat each session, and compare paired results on four gates: time to first substantive audio, barge-in behavior, tool-call accuracy and cost per completed call. Hold the delegated backend model constant, because GPT-Live's backend changes independently [^oai-gpt-live].

Key takeaways:
- Test an upgrade with recorded caller audio replayed through both models, paired by session, not with a few live calls by hand.
- In a full-duplex model, first audio can be a backchannel; measure time to first substantive audio separately.
- Hold the delegated backend model fixed, or an upgrade test measures the backend and not the voice model.
- Score tool calls on three axes: right tool, right moment, right arguments.
- Judge cost per completed call, not per minute, because session time includes silence and backend waits.

## What is confirmed about GPT-Live-1.1, and what is not

This article was written on 2026-11-21 without a readable OpenAI page for GPT-Live-1.1. The announcement URL I tried returned a 404, and the pricing page I could read listed ChatGPT plan prices, not API rates. So this guide makes no claim about what 1.1 changes, what it costs or how fast it is. Take those from OpenAI's release notes and your account's pricing page, then use the method below to find out what they mean for your calls.

What is documented is the architecture you are upgrading. GPT-Live is full duplex: it listens and speaks at the same time and decides many times per second whether to speak, keep listening, pause, interrupt or invoke a tool.[1] For search, deeper reasoning or agentic work it delegates to a separate frontier model, and OpenAI says it will update that backend model as new frontier models ship.[1] That second fact shapes the whole test. A "voice model upgrade" can move two things: the voice model, and the backend behind it. If you do not control the backend, you cannot say which one changed your results.

## Build the recorded-session set

The unit of the test is a recorded session: caller audio, the timestamps of what the caller said, and the tool results your backend returned in the original call. Take them from real calls you are permitted to retain, and check consent and retention before you store audio.

Choose sessions to cover the behavior you will gate on, not to look typical:

- **Plain turns.** Short question, short answer, no tools. These carry the latency baseline.
- **Tool turns.** Calls where the agent looked up an account, booked a slot or took a payment. Keep the original tool arguments and results.
- **Alphanumerics.** Phone numbers, order ids, postcodes, read aloud. OpenAI's gpt-realtime announcement singled out alphanumeric detection as an improvement area, so it is a known weak spot worth a dedicated slice.[2]
- **Interruptions.** Callers who talk over the agent, pause mid-sentence or say "wait, no".
- **Noise and accents.** Background traffic, speakerphone, the languages you actually serve.
- **Long calls.** At least a few past ten minutes, since cost and drift show up late.

Label each session once, by hand or with a reviewed script: when each caller utterance starts and ends, where a barge-in is intended, which tool call is expected and with which arguments, and what counts as a completed outcome. These labels are your ground truth for every candidate model, so do not regenerate them per run.

Two limits are worth stating up front. A replay is open loop: the recorded caller does not react to what the new model says, so a model that answers differently will be talking to a caller who answers the old model. Use replay for the metrics below, which depend on timing and on single turns, and judge whole-conversation quality separately. And replays cost real session time on your own provider account, so budget them as you would any traffic.

## The replay harness

Drive both models with the same audio at real-time pace. Do not send the file faster than speech, because a full-duplex model's behavior depends on the timing of what it hears. The sketch below uses an abstract `session` object; wire it to your own Live API client rather than treating the method names as OpenAI's.

```python
import time, json

def replay(session_factory, recording, backend_model, run_id):
    """Replay one recorded session; return timestamped events."""
    s = session_factory(model=recording["model_under_test"],
                        backend=backend_model)   # pin the backend
    events, t0 = [], time.monotonic()
    s.on_event(lambda e: events.append({
        "t": time.monotonic() - t0, "type": e.type, "data": e.summary()}))

    # Tool results come from the recording, never from live systems.
    s.on_tool_call(lambda call: recording["tool_results"].get(
        key(call), {"error": "unrecorded_call"}))

    for frame in recording["caller_frames"]:      # 20 ms PCM frames
        s.send_audio(frame.pcm)
        time.sleep(frame.duration)                # real-time pace
    s.close()
    return {"run_id": run_id, "session": recording["id"], "events": events,
            "final_usage": s.final_usage()}       # session seconds, tokens
```

Run each session on the current model and on GPT-Live-1.1 from the same machine and region, interleaved so that a network or provider slow period hits both. Repeat each pair three to five times. For outbound SIP sessions, INFRO's GPT-Live guide notes that the sideband replays only the previous three seconds of events and that `session.closed` supplies final usage, so attach your event listener immediately and read usage from the close event, not from a sum of your own timers.[3]

Store one row per event, keyed by session id, run id and model, in a place you can query. If you export spans, pick your own attribute names for voice metrics and map them later: OpenTelemetry's GenAI conventions have moved to their own repository, so check the current names there before you commit to a schema.[4]

## Metric 1: latency to first audio

Define it precisely, because full duplex makes the obvious definition wrong. Measure from the end of the caller's utterance (from your labels, not from the model's own voice-activity events) to the first audio your client receives. Then split two cases:

- **First audio of any kind.** GPT-Live can emit short acknowledgements like "mhmm" or "yeah" while you speak.[1] Those are audio, and they will pull a naive median down.
- **First substantive audio.** The first audio longer than your backchannel cutoff (choose one, for example from the duration distribution of acknowledgements in your own recordings) or the first audio after a tool result.

Report p50, p90 and p99 for both, per slice. Tool turns and plain turns need separate numbers, since delegation to the backend adds waiting that has nothing to do with the voice model. If you already track text first-token time, the same discipline applies; see the guide on [measuring time to first token](https://infro.io/blog/measure-time-to-first-token).

Compare in pairs. For each session, compute the candidate's median minus the current model's median across repeats, then summarize those differences. A fleet-wide percentile hides a regression that lives only in noisy-audio sessions.

## Metric 2: interruption handling

An interruption test needs a scripted barge-in: at a labelled offset, the harness starts playing caller audio while the model is mid-answer. For each scripted barge-in, record:

1. **Stop time.** From the start of caller speech to the last agent audio frame you receive. Audio already buffered in your playback path counts against the user's experience, so measure what reaches your player.
2. **False stops.** Cases where the agent halted for a cough, a "mm-hm" or noise that you labelled as not an interruption. Include these in the set, since the model is deciding many times a second whether to yield.
3. **False holds.** Cases where a real interruption did not stop the agent.
4. **Recovery.** Whether the next answer addresses the interruption, not the sentence that was cut off. Score this with a rubric and a human on a sample.

OpenAI describes older turn-based models as relying on silence for turn detection, which is why a pause or noise could be mistaken for the end of a turn.[1] If your current agent runs on one of those, a 1.1 comparison is also a comparison of turn-taking styles, so expect your own client-side interruption logic to need review. Count a regression in false stops as seriously as a slower stop time: callers experience a model that cuts them off or ignores them as broken, not slow.

## Metric 3: tool-call accuracy

OpenAI's gpt-realtime announcement framed function calling on three axes: calling relevant functions, calling them at the appropriate time and calling them with appropriate arguments.[2] Use the same three, scored against your labels:

| Axis | Pass condition | Typical failure |
| --- | --- | --- |
| Right tool | Expected function named | Wrong lookup, or no call and a spoken guess |
| Right moment | Called after the needed detail was heard, not before | Call fired on a half-spoken order id |
| Right arguments | Normalized arguments equal the label | Digit transposed, wrong date format |

Table: Tool-call scoring axes for a replayed session.

Normalize before comparing: strip spaces and hyphens from phone numbers, canonicalize dates, lowercase enumerations. Then compare exactly. Do not use a judge model for arguments, since an exact check is cheaper and cannot be talked into a pass. Add a fourth count, **unrequested calls**, for side-effecting tools: a call that books or charges when nothing asked for it is worse than a missed one.

Because tool results come from the recording, a candidate that calls a tool with different arguments gets an `unrecorded_call` error. Count that as an argument failure, and review a sample by hand to confirm that the difference was an error and not an equally valid call.

## Metric 4: cost per minute and per completed call

GPT-Live bills session time. INFRO's comparison, which read OpenAI's pricing on 2026-10-27, lists GPT-Live 1 at $0.05 per minute of session, billed per second, with the backend model and tools billed at their own rates; it also notes that active session time includes moments when both sides are silent or the backend is working, and that creating a WebRTC session bills 15 seconds up front.[5] Re-read OpenAI's current page for GPT-Live-1.1 before using any rate, since it may differ from 1 and I could not confirm it.

Compute three numbers per model from the `final_usage` of each replay:

- **Voice cost per call:** session seconds ÷ 60 × the rate on your contract.
- **Backend cost per call:** the tokens and tool charges of delegated work, at their own rates.
- **Cost per completed call:** total cost of all replays of sessions that met the outcome label, divided by the number of those sessions.

The third is the one to decide on. A model that finishes a call faster lowers the session bill and may raise the backend bill; OpenAI's own cost guidance makes that point about backend choice.[5] Reproduce it with your numbers instead of assuming either direction.

> **Example: Illustrative cost arithmetic**
> These figures are invented to show the calculation, not measured. Suppose 100 replayed sessions per model. The current model completes 91 and uses 41,000 session-seconds; the candidate completes 93 and uses 38,500. At an assumed voice rate of $0.05 per minute, voice cost is $34.17 for the current model and $32.08 for the candidate. Add assumed backend costs of $6.10 and $6.90. Cost per completed call is ($34.17 + $6.10) ÷ 91 = $0.44 for the current model and ($32.08 + $6.90) ÷ 93 = $0.42 for the candidate. A shorter session offset a larger backend bill here; with different inputs it would not.

## Hold the backend constant, then decide

GPT-Live's backend can change when OpenAI changes it.[1] Record which backend answered each replay, and where your configuration lets you, pin it for the comparison. If 1.1 ships with a different default backend, run two comparisons: current voice model against 1.1 on the same backend, and 1.1 on its default backend. The first isolates the voice model; the second is what you would actually deploy. If you cannot pin the backend, say so in the report and treat the result as the behavior of a bundle.

Write the gates before the first run. A starting set, with thresholds you should replace with your own:

| Gate | Passes if | Blocks if |
| --- | --- | --- |
| Substantive first audio, p90 | No worse than current plus a margin you chose | Worse on tool turns or noisy slice |
| Barge-in | False stops and false holds not higher | Either count rises |
| Tool calls | Exact-argument rate not lower; zero new unrequested side-effect calls | Any new unrequested side-effect call |
| Cost per completed call | Within your budget band | Above it with no accuracy gain |

Table: Decision gates for a voice model upgrade. Thresholds are yours to set.

Use paired bootstrap resampling over sessions to say whether a difference is more than repeat noise:

```python
import random, statistics as st

def paired_delta_ci(cur, cand, stat=st.median, n=5000, seed=7):
    """cur, cand: per-session values (same order). Returns (delta, lo, hi)."""
    rng = random.Random(seed)
    idx = range(len(cur))
    deltas = []
    for _ in range(n):
        pick = [rng.choice(idx) for _ in idx]
        deltas.append(stat([cand[i] for i in pick]) - stat([cur[i] for i in pick]))
    deltas.sort()
    return (stat(cand) - stat(cur), deltas[int(.025*n)], deltas[int(.975*n)])
```

If the interval for a gate straddles zero, you do not have a result; add sessions or repeats, or accept that the models are equivalent on that gate. For set sizing, see [how many eval cases you need before switching models](https://infro.io/blog/how-many-eval-cases-do-you-need-before-switching-models). After a pass, send a small share of live calls and watch the same four metrics; the article on [replaying production traffic safely](https://infro.io/blog/can-you-replay-production-traffic-to-test-a-new-model-safely) covers the side-effect hazards that matter when real tools are involved.

## Where INFRO fits, and where it does not

The documented INFRO endpoints cover text chat, images, video, speech synthesis and transcription; the pages I read do not describe proxying Live API audio sessions, so the voice leg of this test runs directly against your OpenAI account. INFRO is useful on the text side. If your delegated backend or your scoring runs through chat completions, [experiments](https://infro.io/docs/observability/experiments) replay a dataset across up to six variants, show the request count and estimated provider cost first and stop at a spend cap you set.[6] Datasets, scores and per-request usage can hold your tool-call labels and results beside the rest of your model evaluation. The audio comparison itself stays in your harness.

## Checklist before you switch

- [ ] Recorded sessions cover plain turns, tool turns, alphanumerics, interruptions, noise and long calls, with consent and retention confirmed.
- [ ] Labels for utterance times, barge-ins, expected tool calls and outcomes are fixed and reviewed.
- [ ] Both models replay at real-time pace, interleaved, from the same region.
- [ ] The delegated backend is recorded for every run and pinned where possible.
- [ ] First audio is split into backchannel and substantive.
- [ ] Gates and thresholds were written before the first run.
- [ ] Cost per completed call uses your contract rates and GPT-Live-1.1's current published rate.
- [ ] A live canary with the same four metrics follows the replay.

## Questions

### How many recorded sessions do I need?

Enough that the gap you care about is larger than the noise between repeats of the same session. Run each session three to five times on the current model first to measure that noise, then size the set from it. The article on eval case counts covers the arithmetic.

### Can I use live traffic instead of a replay?

Only as a canary after the replay passes. Live calls differ in every respect except the model, so a latency or accuracy gap is hard to attribute. Replay first, then send a small share of real calls and watch the same four metrics.

## Sources

1. Introducing GPT-Live — OpenAI. https://openai.com/index/introducing-gpt-live/ (accessed 2026-11-21)
2. Introducing gpt-realtime and Realtime API updates for production voice agents — OpenAI. https://openai.com/index/introducing-gpt-realtime/ (accessed 2026-11-21)
3. Outbound SIP on GPT-Live: capacity, cost and what failover can do — INFRO. https://infro.io/blog/outbound-sip-on-gpt-live-capacity-cost-and-what-failover-can-do (accessed 2026-11-21)
4. Generative AI semantic conventions (moved) — OpenTelemetry. https://opentelemetry.io/docs/specs/semconv/gen-ai/ (accessed 2026-11-21)
5. Voice agent APIs compared: GPT-Live, Grok Voice, Voxtral, Gemini — INFRO. https://infro.io/blog/voice-agent-apis-compared-2026 (accessed 2026-11-21)
6. Experiments — Observability — INFRO. https://infro.io/docs/observability/experiments (accessed 2026-11-21)
