# OpenAI Evals shuts down: export your evals and rebuild them in CI

URL: https://infro.io/blog/openai-evals-shutdown-migration
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-27
Last modified: 2026-10-27
Last reviewed: 2026-10-27 (next review due 2026-11-26)
Topic: Evaluation and models
Format: Guide

> OpenAI's Evals platform becomes read-only for existing evals on 31 October 2026 and its dashboard and API are scheduled to shut down on 30 November 2026. OpenAI points to Promptfoo as the migration path and describes recreating evals by hand, not exporting them. Before 31 October, pull eval definitions, runs, per-item results and the dataset files through the Evals and Files APIs, then rebuild each eval as a versioned config in your repository and run it in CI, against every provider you might switch to.

Key takeaways:
- OpenAI announced the Evals deprecation on 3 June 2026: existing evals become read-only on 31 October and the Evals dashboard and API are scheduled to shut down on 30 November 2026.
- OpenAI's own migration guide recreates evals manually in Promptfoo and says it does not require an export feature; plan to export the data yourself.
- The Evals API still lists evals, runs and per-item output items, and the Files API returns your uploaded datasets: pull all of it before evals turn read-only on 31 October.
- Model-graded graders will not reproduce identical scores in another tool; re-validate every judge against a sample of hand-labelled items.
- A suite that lives in your repository and runs in CI against several providers is also your model-retirement and provider-switch test.

OpenAI is closing its hosted Evals product. Existing evals become read-only on 31 October 2026, and the Evals dashboard and API are scheduled to shut down on 30 November 2026[1]. OpenAI's recommended path is Promptfoo, and its migration guide recreates each eval by hand rather than exporting it[2]. If your team uses Evals as a regression gate for prompt or model changes, the work is: pull everything out through the API while it still answers, rebuild the evals as code in your repository, and run them in CI, ideally against more than one provider. This guide covers all three, for the developers doing it and for the leads deciding how much to rebuild.

## The dates and what each one means

| Date | What happens | What you can still do |
| --- | --- | --- |
| 3 June 2026 | Deprecation announced for the Evals platform | Everything |
| 31 October 2026 | Existing evals become read-only | Plan on no new evals, runs or edits; OpenAI does not detail what read-only means for each API call, so export before this date |
| 30 November 2026 | Evals dashboard and API scheduled to shut down | Nothing on the platform |

Table: OpenAI Evals timeline, from the deprecations page read 2026-10-27.

The deprecation page adds that "graders documented for eval workflows are part of this transition", and that fine-tuning timelines are covered separately[1]. Two other products announced the same day share the 30 November date: reusable prompt objects and the `v1/prompts` API, and Agent Builder[1]. If your evals referenced stored prompts, both halves go together; OpenAI's advice for prompts is to move their content into your application code.

The Evals guide itself now carries the same notice, and still recommends Datasets for teams new to evaluation[3]. Datasets is not on the deprecations page as read today.

## What there is to export

An eval on the platform has four parts worth keeping: the definition (its data schema and `testing_criteria`, the graders), the dataset (usually a JSONL file uploaded with purpose `evals`), the runs (which model, which prompt template, the pass and fail counts) and the per-item results. The Evals guide's run object shows the shape: `result_counts`, `per_model_usage`, `per_testing_criteria_results` and a `data_source` that points at a `file_id`[3].

The API reference documents every read you need[4][5]. Run them before 31 October: the deprecation page does not say which calls keep answering once evals are read-only.

| Data | Endpoint |
| --- | --- |
| Every eval | `GET /v1/evals` |
| One eval's definition and graders | `GET /v1/evals/{eval_id}` |
| Its runs | `GET /v1/evals/{eval_id}/runs` |
| One run | `GET /v1/evals/{eval_id}/runs/{run_id}` |
| Each graded item, with `datasource_item` and per-grader `results` | `GET /v1/evals/{eval_id}/runs/{run_id}/output_items` |
| The dataset file | `GET /v1/files/{file_id}/content` |

Table: Evals and Files API reads, from OpenAI's API reference.

Output items can be filtered by `status` (`pass` or `fail`) and paged with `after` and `limit`, and each one carries the source row (`datasource_item`) and a `results` array with each grader's `name`, `passed` and `score`[4]. That is the record you need later to check whether a rebuilt grader agrees with the old one.

### An export script

The script below walks every eval, run and output item and saves them as JSON, plus the dataset files the runs point at. It uses the official Python SDK's list methods, which page automatically. It was written against the documented endpoints and not run against a live account, so try it on one eval first.

```python
import json
import pathlib

from openai import OpenAI

client = OpenAI()  # uses OPENAI_API_KEY for the project that owns the evals
out = pathlib.Path("evals-export")
out.mkdir(exist_ok=True)

def dump(path: pathlib.Path, obj) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    data = obj.model_dump() if hasattr(obj, "model_dump") else obj
    path.write_text(json.dumps(data, indent=2, default=str))

file_ids: set[str] = set()

for ev in client.evals.list():
    base = out / ev.id
    dump(base / "eval.json", client.evals.retrieve(ev.id))
    for run in client.evals.runs.list(ev.id):
        dump(base / "runs" / run.id / "run.json", run)
        source = getattr(run.data_source, "source", None)
        if getattr(source, "type", None) == "file_id":
            file_ids.add(source.id)
        items = [item.model_dump() for item in client.evals.runs.output_items.list(run.id, eval_id=ev.id)]
        dump(base / "runs" / run.id / "output_items.json", items)

for file_id in file_ids:
    content = client.files.content(file_id)
    (out / "files").mkdir(exist_ok=True)
    (out / "files" / f"{file_id}.jsonl").write_bytes(content.read())

print(f"exported {len(file_ids)} dataset files to {out}/")
```

Store the export where your code lives, not in a laptop's downloads folder. The run records include `report_url` links into the dashboard, which is scheduled to shut down on 30 November, so the JSON is the copy that lasts.

## Choosing where the evals go next

OpenAI's guide is candid about the trade: Promptfoo "does not simply reproduce the OpenAI Evals interface in another tool". Evals are defined in a config file and run with the CLI or in CI, graders become assertions and metrics, and "platform features such as dataset review and prompt iteration remain in OpenAI"[2].

| Option | What it is | Where it fits | Where it loses |
| --- | --- | --- | --- |
| Promptfoo | Open-source CLI and library; YAML config, assertions, CI output as JSON, HTML or JUnit XML[6] | Prompt and model regression gates; the path OpenAI documents | No platform-hosted dataset review; thresholds over a whole run need a short script[6] |
| Inspect | Open-source evaluation framework from the UK AI Security Institute and Meridian Labs; tasks made of a dataset, a solver and a scorer[7] | Agentic and multi-step evals, written in Python | More code per eval than a YAML assertion |
| openai/evals repository | The open-source framework and benchmark registry behind the original Evals; not archived, last pushed April 2026[8] | Teams already invested in its registry format | Little recent maintenance; you own it |
| Your own test runner | pytest, Vitest or similar, calling the model and asserting | Small suites with exact-match checks | You rebuild grading, reporting and judge prompts yourself |

Table: Where OpenAI Evals workloads can move, from each tool's own documentation.

For most teams with string-check and model-graded evals, the shortest path is the one OpenAI documents. Promptfoo's assertion list covers the common graders: `equals`, `contains`, `regex`, `is-json` and `javascript` or `python` for custom checks; `similar`, `llm-rubric` and `g-eval` for model-graded ones; and `cost` and `latency` thresholds[9]. Inspect is the better fit when an eval is really an agent task with tools.

## Rebuild the suite so it runs in CI against several providers

The migration is a chance to fix the main weakness of a dashboard-hosted eval: it tested one provider, in one place, when someone remembered to run it.

1. **Recreate each eval from the export.** Turn the dataset JSONL into test cases and each `testing_criteria` entry into an assertion. A `string_check` with `eq` becomes `equals`; a label or score model grader becomes `llm-rubric` with the same rubric text. The API reference lists five grader types to map: string check, text similarity, label model, score model and Python[4].
2. **Validate every rebuilt grader.** OpenAI warns that "similarity-based scoring may not produce identical numerical results across systems" and that recreated LLM-as-a-judge graders should be validated before you rely on them[2]. Run the new suite on the exported items and compare pass and fail per item with the old `results`. Where they disagree, read the item before you change the threshold.
3. **Set the judge model explicitly.** In Promptfoo, gateway settings under `providers` do not configure the grader for model-graded assertions; set it separately under `defaultTest.options.provider`[10]. Pin the judge to a dated model so a judge upgrade cannot move your scores.
4. **Add providers, not just models.** Promptfoo compares models by listing several `providers`; Anthropic models use `anthropic:messages:<model>`[11], and the OpenAI provider takes an `apiBaseUrl` for any OpenAI-compatible endpoint[10]. Run the same tests on your current model and the two you would switch to.
5. **Fail the build on regressions.** `--fail-on-error` fails on any test failure, `-o results.junit.xml` feeds CI test viewers, and `--tag` records the commit; a pass-rate threshold is a few lines of `jq` over the JSON output[6].

```yaml
prompts:
  - file://prompts/ticket-triage.txt
providers:
  - id: openai:chat:gpt-6-luna
  - id: anthropic:messages:claude-sonnet-5.5
defaultTest:
  options:
    provider: openai:chat:gpt-6-sol   # the judge, pinned separately
tests: file://evals-export/converted/ticket-triage.yaml
```

> **Example: What a nightly multi-provider run costs, at list rates**
> A suite of 400 test cases averaging 1,500 input and 300 output tokens runs nightly on two models, with an LLM judge reading 2,000 tokens and writing 100 per graded case. On GPT-6 Luna ($0.10 input, $0.50 output per million) the candidate pass costs 400 × (1,500 × $0.10 + 300 × $0.50) / 1M = $0.12. On Claude Sonnet 5.5 ($2 and $10) it costs 400 × (1,500 × $2 + 300 × $10) / 1M = $2.40. A GPT-6 Sol judge ($2 and $10) on all 800 graded outputs costs 800 × (2,000 × $2 + 100 × $10) / 1M = $4.00. About $6.52 a night, or roughly $196 over 30 nights, and most of it is the judge. Token counts are assumptions; read yours from the first run.

The arithmetic matters for one decision: whether every commit runs the full suite or only a smoke set, with the full suite nightly and before any model change.

## What to ask your team this week

- Which evals are actually used as a gate before a release, and which were one-off experiments? Migrate the first group first.
- Who owns the export, and where will it be stored?
- Which graders are model-graded, and have their new versions been checked against the old results?
- Which providers will the suite run against, and does a model retirement or provider outage now have a test that runs before the switch?

## Running the suite through INFRO

INFRO is a control plane in front of the provider accounts you already hold. It is not an eval harness: it stores no datasets, runs no test cases and keeps no grader definitions, so the export and the rebuilt suite above are yours to keep.

What it adds is the multi-provider half of step 4. INFRO's `/v1/chat/completions` takes the request shape the OpenAI SDKs send, so a Promptfoo `openai:chat` provider with `apiBaseUrl: https://api.infro.io/v1` and an INFRO gateway key can call `openai/gpt-6-luna`, `anthropic/claude-sonnet-5.5` and `google/gemini-3.8-flash` on your own connections, each billed by its provider to you. A separate gateway key for CI keeps eval traffic in its own usage line and under its own spend limit, checked before each call. In production, eval monitors score a share of served chat answers in the background against your rubric with a decision model you connect, recorded per request and summarised per served model; that watches live traffic between releases and does not replace a regression suite run before one.

## Questions

### When does OpenAI Evals shut down?

OpenAI's deprecations page lists three dates: deprecation announced on 3 June 2026, existing evals becoming read-only on 31 October 2026, and the Evals dashboard and API scheduled to shut down on 30 November 2026. Graders documented for eval workflows are part of the same transition.

### Can I export my evals from OpenAI?

OpenAI's migration guide recreates evals by hand and says it does not require an Evals export feature. You can still read your data through the API while it runs: GET /evals lists evals, GET /evals/{eval_id}/runs lists runs, GET /evals/{eval_id}/runs/{run_id}/output_items returns each graded item, and GET /files/{file_id}/content returns the dataset files you uploaded. OpenAI does not say which API calls keep working once evals are read-only on 31 October, so export before that date rather than counting on November.

### What does OpenAI recommend instead of Evals?

OpenAI recommends Promptfoo, an open-source CLI and library, and publishes a cookbook on recreating an eval as a promptfooconfig.yaml and running it locally or in CI. Other harnesses exist, such as Inspect from the UK AI Security Institute; choose by how your evals are written and where they must run.

### Will my scores be the same after migrating?

Not necessarily. OpenAI's own guide warns that similarity-based scoring may not produce identical numbers across systems and that any recreated grader, especially an LLM-as-a-judge grader, should be validated before you rely on it for regression decisions. Compare old and new scores on the same items before you trust a threshold.

### Is the open-source openai/evals repository affected?

The deprecation page covers the Evals platform: the dashboard and the Evals API. The openai/evals repository on GitHub was not archived when checked on 27 October 2026, though its last push was in April 2026 and its README points to the dashboard. Treat it as a framework you would maintain yourself.

## Sources

1. Deprecations — OpenAI. https://developers.openai.com/api/docs/deprecations (accessed 2026-10-27)
2. Moving from OpenAI Evals to Promptfoo — OpenAI. https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo (accessed 2026-10-27)
3. Working with evals — OpenAI. https://developers.openai.com/api/docs/guides/evals (accessed 2026-10-27)
4. Evals API reference — OpenAI. https://developers.openai.com/api/reference/resources/evals (accessed 2026-10-27)
5. Files API reference — OpenAI. https://developers.openai.com/api/reference/resources/files (accessed 2026-10-27)
6. CI/CD integration — Promptfoo. https://www.promptfoo.dev/docs/integrations/ci-cd/ (accessed 2026-10-27)
7. Inspect — UK AI Security Institute. https://inspect.aisi.org.uk/ (accessed 2026-10-27)
8. openai/evals — OpenAI on GitHub. https://github.com/openai/evals (accessed 2026-10-27)
9. Assertions and metrics — Promptfoo. https://www.promptfoo.dev/docs/configuration/expected-outputs/ (accessed 2026-10-27)
10. OpenAI provider — Promptfoo. https://www.promptfoo.dev/docs/providers/openai/ (accessed 2026-10-27)
11. Anthropic provider — Promptfoo. https://www.promptfoo.dev/docs/providers/anthropic/ (accessed 2026-10-27)