Skip to content

Evaluation and modelsMethodology

Testing GPT-Live-1.1: a recorded-session method for voice agents

A voice model upgrade changes how a call sounds, not only what it says. This method replays recorded sessions against the current and candidate GPT-Live versions and scores four things: latency to first audio, interruption handling, tool-call accuracy and cost per completed call.

By INFRO Editorial

Published

10 min read

A single amber path on a dark ground forking into two glass branches of slightly different brightness

In short

Replay the same recorded caller audio through your current GPT-Live model and GPT-Live-1.1, repeat each session, and compare paired results on four gates: time to first substantive audio, barge-in behavior, tool-call accuracy and cost per completed call. Hold the delegated backend model constant, because GPT-Live's backend changes independently 1.

Key takeaways

  • Test an upgrade with recorded caller audio replayed through both models, paired by session, not with a few live calls by hand.
  • In a full-duplex model, first audio can be a backchannel; measure time to first substantive audio separately.
  • Hold the delegated backend model fixed, or an upgrade test measures the backend and not the voice model.
  • Score tool calls on three axes: right tool, right moment, right arguments.
  • Judge cost per completed call, not per minute, because session time includes silence and backend waits.

What is confirmed about GPT-Live-1.1, and what is not

This article was written on 2026-11-21 without a readable OpenAI page for GPT-Live-1.1. The announcement URL I tried returned a 404, and the pricing page I could read listed ChatGPT plan prices, not API rates. So this guide makes no claim about what 1.1 changes, what it costs or how fast it is. Take those from OpenAI's release notes and your account's pricing page, then use the method below to find out what they mean for your calls.

What is documented is the architecture you are upgrading. GPT-Live is full duplex: it listens and speaks at the same time and decides many times per second whether to speak, keep listening, pause, interrupt or invoke a tool.1 For search, deeper reasoning or agentic work it delegates to a separate frontier model, and OpenAI says it will update that backend model as new frontier models ship.1 That second fact shapes the whole test. A "voice model upgrade" can move two things: the voice model, and the backend behind it. If you do not control the backend, you cannot say which one changed your results.

Build the recorded-session set

The unit of the test is a recorded session: caller audio, the timestamps of what the caller said, and the tool results your backend returned in the original call. Take them from real calls you are permitted to retain, and check consent and retention before you store audio.

Choose sessions to cover the behavior you will gate on, not to look typical:

  • Plain turns. Short question, short answer, no tools. These carry the latency baseline.
  • Tool turns. Calls where the agent looked up an account, booked a slot or took a payment. Keep the original tool arguments and results.
  • Alphanumerics. Phone numbers, order ids, postcodes, read aloud. OpenAI's gpt-realtime announcement singled out alphanumeric detection as an improvement area, so it is a known weak spot worth a dedicated slice.2
  • Interruptions. Callers who talk over the agent, pause mid-sentence or say "wait, no".
  • Noise and accents. Background traffic, speakerphone, the languages you actually serve.
  • Long calls. At least a few past ten minutes, since cost and drift show up late.

Label each session once, by hand or with a reviewed script: when each caller utterance starts and ends, where a barge-in is intended, which tool call is expected and with which arguments, and what counts as a completed outcome. These labels are your ground truth for every candidate model, so do not regenerate them per run.

Two limits are worth stating up front. A replay is open loop: the recorded caller does not react to what the new model says, so a model that answers differently will be talking to a caller who answers the old model. Use replay for the metrics below, which depend on timing and on single turns, and judge whole-conversation quality separately. And replays cost real session time on your own provider account, so budget them as you would any traffic.

The replay harness

Drive both models with the same audio at real-time pace. Do not send the file faster than speech, because a full-duplex model's behavior depends on the timing of what it hears. The sketch below uses an abstract session object; wire it to your own Live API client rather than treating the method names as OpenAI's.

replay.py
import time, json

def replay(session_factory, recording, backend_model, run_id):
    """Replay one recorded session; return timestamped events."""
    s = session_factory(model=recording["model_under_test"],
                        backend=backend_model)   # pin the backend
    events, t0 = [], time.monotonic()
    s.on_event(lambda e: events.append({
        "t": time.monotonic() - t0, "type": e.type, "data": e.summary()}))

    # Tool results come from the recording, never from live systems.
    s.on_tool_call(lambda call: recording["tool_results"].get(
        key(call), {"error": "unrecorded_call"}))

    for frame in recording["caller_frames"]:      # 20 ms PCM frames
        s.send_audio(frame.pcm)
        time.sleep(frame.duration)                # real-time pace
    s.close()
    return {"run_id": run_id, "session": recording["id"], "events": events,
            "final_usage": s.final_usage()}       # session seconds, tokens

Run each session on the current model and on GPT-Live-1.1 from the same machine and region, interleaved so that a network or provider slow period hits both. Repeat each pair three to five times. For outbound SIP sessions, INFRO's GPT-Live guide notes that the sideband replays only the previous three seconds of events and that session.closed supplies final usage, so attach your event listener immediately and read usage from the close event, not from a sum of your own timers.3

Store one row per event, keyed by session id, run id and model, in a place you can query. If you export spans, pick your own attribute names for voice metrics and map them later: OpenTelemetry's GenAI conventions have moved to their own repository, so check the current names there before you commit to a schema.4

Metric 1: latency to first audio

Define it precisely, because full duplex makes the obvious definition wrong. Measure from the end of the caller's utterance (from your labels, not from the model's own voice-activity events) to the first audio your client receives. Then split two cases:

  • First audio of any kind. GPT-Live can emit short acknowledgements like "mhmm" or "yeah" while you speak.1 Those are audio, and they will pull a naive median down.
  • First substantive audio. The first audio longer than your backchannel cutoff (choose one, for example from the duration distribution of acknowledgements in your own recordings) or the first audio after a tool result.

Report p50, p90 and p99 for both, per slice. Tool turns and plain turns need separate numbers, since delegation to the backend adds waiting that has nothing to do with the voice model. If you already track text first-token time, the same discipline applies; see the guide on measuring time to first token.

Compare in pairs. For each session, compute the candidate's median minus the current model's median across repeats, then summarize those differences. A fleet-wide percentile hides a regression that lives only in noisy-audio sessions.

Metric 2: interruption handling

An interruption test needs a scripted barge-in: at a labelled offset, the harness starts playing caller audio while the model is mid-answer. For each scripted barge-in, record:

  1. Stop time. From the start of caller speech to the last agent audio frame you receive. Audio already buffered in your playback path counts against the user's experience, so measure what reaches your player.
  2. False stops. Cases where the agent halted for a cough, a "mm-hm" or noise that you labelled as not an interruption. Include these in the set, since the model is deciding many times a second whether to yield.
  3. False holds. Cases where a real interruption did not stop the agent.
  4. Recovery. Whether the next answer addresses the interruption, not the sentence that was cut off. Score this with a rubric and a human on a sample.

OpenAI describes older turn-based models as relying on silence for turn detection, which is why a pause or noise could be mistaken for the end of a turn.1 If your current agent runs on one of those, a 1.1 comparison is also a comparison of turn-taking styles, so expect your own client-side interruption logic to need review. Count a regression in false stops as seriously as a slower stop time: callers experience a model that cuts them off or ignores them as broken, not slow.

Metric 3: tool-call accuracy

OpenAI's gpt-realtime announcement framed function calling on three axes: calling relevant functions, calling them at the appropriate time and calling them with appropriate arguments.2 Use the same three, scored against your labels:

AxisPass conditionTypical failure
Right toolExpected function namedWrong lookup, or no call and a spoken guess
Right momentCalled after the needed detail was heard, not beforeCall fired on a half-spoken order id
Right argumentsNormalized arguments equal the labelDigit transposed, wrong date format
Tool-call scoring axes for a replayed session.

Normalize before comparing: strip spaces and hyphens from phone numbers, canonicalize dates, lowercase enumerations. Then compare exactly. Do not use a judge model for arguments, since an exact check is cheaper and cannot be talked into a pass. Add a fourth count, unrequested calls, for side-effecting tools: a call that books or charges when nothing asked for it is worse than a missed one.

Because tool results come from the recording, a candidate that calls a tool with different arguments gets an unrecorded_call error. Count that as an argument failure, and review a sample by hand to confirm that the difference was an error and not an equally valid call.

Metric 4: cost per minute and per completed call

GPT-Live bills session time. INFRO's comparison, which read OpenAI's pricing on 2026-10-27, lists GPT-Live 1 at $0.05 per minute of session, billed per second, with the backend model and tools billed at their own rates; it also notes that active session time includes moments when both sides are silent or the backend is working, and that creating a WebRTC session bills 15 seconds up front.5 Re-read OpenAI's current page for GPT-Live-1.1 before using any rate, since it may differ from 1 and I could not confirm it.

Compute three numbers per model from the final_usage of each replay:

  • Voice cost per call: session seconds ÷ 60 × the rate on your contract.
  • Backend cost per call: the tokens and tool charges of delegated work, at their own rates.
  • Cost per completed call: total cost of all replays of sessions that met the outcome label, divided by the number of those sessions.

The third is the one to decide on. A model that finishes a call faster lowers the session bill and may raise the backend bill; OpenAI's own cost guidance makes that point about backend choice.5 Reproduce it with your numbers instead of assuming either direction.

Hold the backend constant, then decide

GPT-Live's backend can change when OpenAI changes it.1 Record which backend answered each replay, and where your configuration lets you, pin it for the comparison. If 1.1 ships with a different default backend, run two comparisons: current voice model against 1.1 on the same backend, and 1.1 on its default backend. The first isolates the voice model; the second is what you would actually deploy. If you cannot pin the backend, say so in the report and treat the result as the behavior of a bundle.

Write the gates before the first run. A starting set, with thresholds you should replace with your own:

GatePasses ifBlocks if
Substantive first audio, p90No worse than current plus a margin you choseWorse on tool turns or noisy slice
Barge-inFalse stops and false holds not higherEither count rises
Tool callsExact-argument rate not lower; zero new unrequested side-effect callsAny new unrequested side-effect call
Cost per completed callWithin your budget bandAbove it with no accuracy gain
Decision gates for a voice model upgrade. Thresholds are yours to set.

Use paired bootstrap resampling over sessions to say whether a difference is more than repeat noise:

paired_bootstrap.py
import random, statistics as st

def paired_delta_ci(cur, cand, stat=st.median, n=5000, seed=7):
    """cur, cand: per-session values (same order). Returns (delta, lo, hi)."""
    rng = random.Random(seed)
    idx = range(len(cur))
    deltas = []
    for _ in range(n):
        pick = [rng.choice(idx) for _ in idx]
        deltas.append(stat([cand[i] for i in pick]) - stat([cur[i] for i in pick]))
    deltas.sort()
    return (stat(cand) - stat(cur), deltas[int(.025*n)], deltas[int(.975*n)])

If the interval for a gate straddles zero, you do not have a result; add sessions or repeats, or accept that the models are equivalent on that gate. For set sizing, see how many eval cases you need before switching models. After a pass, send a small share of live calls and watch the same four metrics; the article on replaying production traffic safely covers the side-effect hazards that matter when real tools are involved.

Where INFRO fits, and where it does not

The documented INFRO endpoints cover text chat, images, video, speech synthesis and transcription; the pages I read do not describe proxying Live API audio sessions, so the voice leg of this test runs directly against your OpenAI account. INFRO is useful on the text side. If your delegated backend or your scoring runs through chat completions, experiments (opens in a new tab) replay a dataset across up to six variants, show the request count and estimated provider cost first and stop at a spend cap you set.6 Datasets, scores and per-request usage can hold your tool-call labels and results beside the rest of your model evaluation. The audio comparison itself stays in your harness.

Checklist before you switch

  • Recorded sessions cover plain turns, tool turns, alphanumerics, interruptions, noise and long calls, with consent and retention confirmed.
  • Labels for utterance times, barge-ins, expected tool calls and outcomes are fixed and reviewed.
  • Both models replay at real-time pace, interleaved, from the same region.
  • The delegated backend is recorded for every run and pinned where possible.
  • First audio is split into backchannel and substantive.
  • Gates and thresholds were written before the first run.
  • Cost per completed call uses your contract rates and GPT-Live-1.1's current published rate.
  • A live canary with the same four metrics follows the replay.

Questions readers ask

How many recorded sessions do I need?
Enough that the gap you care about is larger than the noise between repeats of the same session. Run each session three to five times on the current model first to measure that noise, then size the set from it. The article on eval case counts covers the arithmetic.
Can I use live traffic instead of a replay?
Only as a canary after the replay passes. Live calls differ in every respect except the model, so a latency or accuracy gap is hard to attribute. Replay first, then send a small share of real calls and watch the same four metrics.

From the team that builds it

Where this shows up in INFRO

INFRO is a control plane in front of the provider accounts you already hold. The capabilities below are the parts of it this article touches, each with the documentation that defines its behaviour.

  • Experiments

    Available with a plan

    A dataset run through the gateway across up to six variants (a model or route, a prompt version, settings), judged by your evaluators and compared side by side, with the request count and estimated provider cost shown before it starts and a spend cap it stops at. On every plan.

  • Datasets

    Available with a plan

    Inputs and expected outputs built from logged requests in one click, uploaded as JSONL or CSV, or sent from code, kept until you delete them. On every plan.

  • Scores and end-user feedback

    Available with a plan

    Scores on a request or a run from your code, from eval monitors and from members in the console, and feedback from the people reading your answers: each response carries a signed token a browser can send with a thumbs up or down, without an API key. Scores do not count against the request allowance. On every plan.

  • Usage and cost per request

    Available with a plan

    Every request records its model, latency, token or unit counts, and estimated provider cost, queryable by project and key.

    Read the documentation for Usage and cost per request

INFRO accounts are self-serve. Provider usage is billed to you by your providers; INFRO charges a software subscription and takes 0% of model spend. See plans.

Sources

Numbered in order of first citation. Each was opened and checked against the claim on the date shown.

  1. 1.Introducing GPT-Live (opens in a new tab) — OpenAI. Accessed Nov 21, 2026.
  2. 2.Introducing gpt-realtime and Realtime API updates for production voice agents (opens in a new tab) — OpenAI. Accessed Nov 21, 2026.
  3. 3.Outbound SIP on GPT-Live: capacity, cost and what failover can do (opens in a new tab) — INFRO. Accessed Nov 21, 2026.
  4. 4.Generative AI semantic conventions (moved) (opens in a new tab) — OpenTelemetry. Accessed Nov 21, 2026.
  5. 5.Voice agent APIs compared: GPT-Live, Grok Voice, Voxtral, Gemini (opens in a new tab) — INFRO. Accessed Nov 21, 2026.
  6. 6.Experiments — Observability (opens in a new tab) — INFRO. Accessed Nov 21, 2026.

About this article

Read on 2026-11-21: OpenAI's GPT-Live launch post, the gpt-realtime announcement, OpenTelemetry's GenAI conventions page, INFRO's experiments docs and two INFRO articles on GPT-Live. OpenAI's page for GPT-Live-1.1 returned 404 and the pricing page returned ChatGPT plan pricing, so 1.1's rates, latency and changes are left out as unconfirmed.

Last checked against its sources on . The next review is due by . No separate named reviewer is recorded for this article.

How articles are written, sourced and corrected is set out in the editorial standards.