OpenAI Evals Shutdown: What to Migrate Before November 30

The OpenAI Evals shutdown is now measured in days, not months. OpenAI's own documentation states that the Evals platform becomes read-only for existing users on October 31, 2026 and is scheduled to shut down on November 30, 2026. If your team stores evaluation sets, graders, or run history in Evals, you have a short window to export what matters, pick a new home, and prove your quality checks still work. This post lays out the timeline, the official Datasets path, the third-party options OpenAI itself points to, and a practical migration plan for the remaining weeks. Source

The timeline and what read-only actually means

OpenAI posted the deprecation notice in its community forum on July 2, 2026, and the same dates now appear at the top of both the Evals guide and the Datasets guide on the developer site. The transition has two phases. Until October 31, existing evals content remains fully available. From October 31 the platform goes read-only: you can view and export, but not create or run. On November 30 the platform shuts down entirely. Source

Read-only is the dangerous phase. It feels safe because nothing has been deleted yet, but every day past October 31 is a day you cannot run a regression check against your new setup using the old system as reference. The work that matters, exporting data and rebuilding graders, has to be finished before the read-only date to be verified properly, not just before shutdown.

What Evals actually did for teams

Evals gave teams a structured three-step loop: describe the task as an eval, run it with test inputs (a prompt plus input data), then analyze results and iterate on the prompt. OpenAI's own guide compares the process to behavior-driven development, where you specify expected behavior before you build. For teams shipping LLM features, this turned prompt quality from vibes into a repeatable check: a stored dataset of inputs and ground truth, graders that score outputs, and a run history that shows whether a prompt change helped or hurt. Source

That loop is what you are really migrating. The export is the easy part. The hard part is preserving the habit: every prompt change runs against a known set of cases, and the result is a number you can compare to last week. Any replacement that loses that property is a downgrade even if the UI looks nicer. If you are building or validating an AI product, this is the same discipline we described in our post on AI MVP validation, where a small honest eval set beats a large vague one.

The official path: Datasets

OpenAI's recommended destination for most users is Datasets, a feature of the OpenAI platform designed as a quick way to get started with evals and test prompts. The getting-started guide walks through creating a dataset from the Datasets tab in the dashboard, uploading a CSV with input columns and ground truth columns, editing rows in a visual interface, and then building prompts whose graders can reference those columns. OpenAI recommends treating a dataset as a living space that grows as you discover edge cases. Source

Datasets is a good fit if your evals are prompt-iteration tools: you want to tweak a prompt in the dashboard, run it against a CSV of cases, and eyeball or grade results. It is deliberately lighter than Evals. The same documentation is explicit about the boundary: if you need evaluation against external models, want to drive eval runs through the API, or want to run evaluations at larger scale, Datasets is not the tool. Those were Evals' strengths, and they are exactly the capabilities that disappear on November 30. Source

So the decision is simple to state. Dashboard-driven prompt testing moves to Datasets. Programmatic, API-driven, external-model, or CI-integrated evaluation needs a third-party home.

The third-party paths OpenAI points to

OpenAI's cookbook now includes a guide titled "Moving from OpenAI Evals to Promptfoo", which winds down the Evals product and recommends Promptfoo for continuing and extending evaluation workflows. Langfuse publishes a similar migration guide. Both are open-source evaluation frameworks that run your cases through your own infrastructure, support graders and assertions, and integrate with CI, which is precisely the gap Datasets leaves open. Source

A caveat from the field: the July community thread about the shutdown shows at least one user unable to find the "export as runnable Promptfoo config" button the cookbook describes, which suggests the export path has rough edges. Budget time for manual reconstruction of graders, not just a one-click export, and verify every migrated grader against known outputs before you trust it. Source

A migration plan for the remaining weeks

With read-only starting October 31 and shutdown on November 30, work backwards from this week.

First, inventory and export now. List every eval you have: its dataset, its graders, and the last few runs you care about as reference. Export the raw data while the platform is fully writable, and store it somewhere boring and durable. Do this even for evals you think are dead; reconstruction from memory in December is worse than a five-minute export today.

Second, triage into three buckets. Prompt-iteration evals go to Datasets. API-driven or CI-integrated evals go to Promptfoo, Langfuse, or a similar framework. Anything nobody has run in six months is a candidate for archive, not migration. The cookbook's Promptfoo guide and OpenAI's Datasets guide give you the exact mechanics for the first two buckets. Source

Third, rebuild graders and prove parity. For each migrated eval, run the same inputs through the new grader and check that scores land close to the old system's results on reference runs. A grader that silently changed meaning is worse than no grader, because it manufactures false confidence in every future prompt change.

Fourth, wire the new evals into your delivery flow. An eval that lives only in a dashboard gets skipped under deadline pressure. The goal is the same regression-check habit Evals enabled: prompt changes trigger a run, the run produces a comparable score, and a drop blocks the merge or at least demands a human look. Our AI agent observability checklist covers how this fits into the wider monitoring picture for agent systems.

What this signals for agent teams

The Evals shutdown is not OpenAI leaving evaluation; it is OpenAI narrowing its evaluation product to the dashboard use case and ceding the engineering-grade segment to open tooling. For teams running agents in production, the lesson generalizes: your eval sets, graders, and run history are operational assets, and they should live in formats and systems you control, not only inside a vendor's platform. Treat the next few weeks as a forced but useful audit of that principle. Source

If your team needs help moving evaluation pipelines off Evals and into a CI-friendly setup, this is the kind of AgentOps work we do at Codesprint, alongside the testing discipline in our AI agent safety testing guide.