OpenAI Evals shuts down: export your evals and rebuild them in CI

OpenAI is closing its hosted Evals product. Existing evals become read-only on 31 October 2026, and the Evals dashboard and API are scheduled to shut down on 30 November 2026¹. OpenAI's recommended path is Promptfoo, and its migration guide recreates each eval by hand rather than exporting it². If your team uses Evals as a regression gate for prompt or model changes, the work is: pull everything out through the API while it still answers, rebuild the evals as code in your repository, and run them in CI, ideally against more than one provider. This guide covers all three, for the developers doing it and for the leads deciding how much to rebuild.

The dates and what each one means

The deprecation page adds that "graders documented for eval workflows are part of this transition", and that fine-tuning timelines are covered separately¹. Two other products announced the same day share the 30 November date: reusable prompt objects and the v1/prompts API, and Agent Builder¹. If your evals referenced stored prompts, both halves go together; OpenAI's advice for prompts is to move their content into your application code.

The Evals guide itself now carries the same notice, and still recommends Datasets for teams new to evaluation³. Datasets is not on the deprecations page as read today.

What there is to export

An eval on the platform has four parts worth keeping: the definition (its data schema and testing_criteria, the graders), the dataset (usually a JSONL file uploaded with purpose evals), the runs (which model, which prompt template, the pass and fail counts) and the per-item results. The Evals guide's run object shows the shape: result_counts, per_model_usage, per_testing_criteria_results and a data_source that points at a file_id³.

The API reference documents every read you need⁴⁵. Run them before 31 October: the deprecation page does not say which calls keep answering once evals are read-only.

Output items can be filtered by status (pass or fail) and paged with after and limit, and each one carries the source row (datasource_item) and a results array with each grader's name, passed and score⁴. That is the record you need later to check whether a rebuilt grader agrees with the old one.

An export script

The script below walks every eval, run and output item and saves them as JSON, plus the dataset files the runs point at. It uses the official Python SDK's list methods, which page automatically. It was written against the documented endpoints and not run against a live account, so try it on one eval first.

import json
import pathlib

from openai import OpenAI

client = OpenAI()  # uses OPENAI_API_KEY for the project that owns the evals
out = pathlib.Path("evals-export")
out.mkdir(exist_ok=True)

def dump(path: pathlib.Path, obj) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    data = obj.model_dump() if hasattr(obj, "model_dump") else obj
    path.write_text(json.dumps(data, indent=2, default=str))

file_ids: set[str] = set()

for ev in client.evals.list():
    base = out / ev.id
    dump(base / "eval.json", client.evals.retrieve(ev.id))
    for run in client.evals.runs.list(ev.id):
        dump(base / "runs" / run.id / "run.json", run)
        source = getattr(run.data_source, "source", None)
        if getattr(source, "type", None) == "file_id":
            file_ids.add(source.id)
        items = [item.model_dump() for item in client.evals.runs.output_items.list(run.id, eval_id=ev.id)]
        dump(base / "runs" / run.id / "output_items.json", items)

for file_id in file_ids:
    content = client.files.content(file_id)
    (out / "files").mkdir(exist_ok=True)
    (out / "files" / f"{file_id}.jsonl").write_bytes(content.read())

print(f"exported {len(file_ids)} dataset files to {out}/")

Store the export where your code lives, not in a laptop's downloads folder. The run records include report_url links into the dashboard, which is scheduled to shut down on 30 November, so the JSON is the copy that lasts.

Choosing where the evals go next

OpenAI's guide is candid about the trade: Promptfoo "does not simply reproduce the OpenAI Evals interface in another tool". Evals are defined in a config file and run with the CLI or in CI, graders become assertions and metrics, and "platform features such as dataset review and prompt iteration remain in OpenAI"².

For most teams with string-check and model-graded evals, the shortest path is the one OpenAI documents. Promptfoo's assertion list covers the common graders: equals, contains, regex, is-json and javascript or python for custom checks; similar, llm-rubric and g-eval for model-graded ones; and cost and latency thresholds⁹. Inspect is the better fit when an eval is really an agent task with tools.

Rebuild the suite so it runs in CI against several providers

The migration is a chance to fix the main weakness of a dashboard-hosted eval: it tested one provider, in one place, when someone remembered to run it.

1. Recreate each eval from the export. Turn the dataset JSONL into test cases and each testing_criteria entry into an assertion. A string_check with eq becomes equals; a label or score model grader becomes llm-rubric with the same rubric text. The API reference lists five grader types to map: string check, text similarity, label model, score model and Python⁴.
2. Validate every rebuilt grader. OpenAI warns that "similarity-based scoring may not produce identical numerical results across systems" and that recreated LLM-as-a-judge graders should be validated before you rely on them². Run the new suite on the exported items and compare pass and fail per item with the old results. Where they disagree, read the item before you change the threshold.
3. Set the judge model explicitly. In Promptfoo, gateway settings under providers do not configure the grader for model-graded assertions; set it separately under defaultTest.options.provider¹⁰. Pin the judge to a dated model so a judge upgrade cannot move your scores.
4. Add providers, not just models. Promptfoo compares models by listing several providers; Anthropic models use anthropic:messages:<model>¹¹, and the OpenAI provider takes an apiBaseUrl for any OpenAI-compatible endpoint¹⁰. Run the same tests on your current model and the two you would switch to.
5. Fail the build on regressions. --fail-on-error fails on any test failure, -o results.junit.xml feeds CI test viewers, and --tag records the commit; a pass-rate threshold is a few lines of jq over the JSON output⁶.

prompts:
  - file://prompts/ticket-triage.txt
providers:
  - id: openai:chat:gpt-6-luna
  - id: anthropic:messages:claude-sonnet-5.5
defaultTest:
  options:
    provider: openai:chat:gpt-6-sol   # the judge, pinned separately
tests: file://evals-export/converted/ticket-triage.yaml

The arithmetic matters for one decision: whether every commit runs the full suite or only a smoke set, with the full suite nightly and before any model change.

What to ask your team this week

• Which evals are actually used as a gate before a release, and which were one-off experiments? Migrate the first group first.
• Who owns the export, and where will it be stored?
• Which graders are model-graded, and have their new versions been checked against the old results?
• Which providers will the suite run against, and does a model retirement or provider outage now have a test that runs before the switch?

Running the suite through INFRO

INFRO is a control plane in front of the provider accounts you already hold. It is not an eval harness: it stores no datasets, runs no test cases and keeps no grader definitions, so the export and the rebuilt suite above are yours to keep.

What it adds is the multi-provider half of step 4. INFRO's /v1/chat/completions takes the request shape the OpenAI SDKs send, so a Promptfoo openai:chat provider with apiBaseUrl: https://api.infro.io/v1 and an INFRO gateway key can call openai/gpt-6-luna, anthropic/claude-sonnet-5.5 and google/gemini-3.8-flash on your own connections, each billed by its provider to you. A separate gateway key for CI keeps eval traffic in its own usage line and under its own spend limit, checked before each call. In production, eval monitors score a share of served chat answers in the background against your rubric with a decision model you connect, recorded per request and summarised per served model; that watches live traffic between releases and does not replace a regression suite run before one.