Portable AI evaluations: what to export before switching platforms
An AI evaluation is sufficiently portable when your team can understand it outside the original platform and rerun it with reasonable effort. For complex tasks, exporting questions and answers is insufficient. You also need expectations, scoring logic, versions and results for each test case.
The platform can run and analyse evaluations. Keep the business acceptance criteria in a form you control as well.
Why this is a practical issue now
OpenAI announced the retirement of its existing Evals platform on 3 June 2026. Existing evals are scheduled to become read-only on 31 October 2026; the dashboard and API are scheduled to shut down on 30 November 2026. Associated graders are part of the transition. The official deprecation documentation points to Promptfoo as a migration path. Source checked on 22 October 2026.
This announcement concerns a specific product. It is also a practical reason to inspect your evaluation assets: which information exists only in a provider dashboard?
Retention limits matter too. Microsoft's documentation for single-response tests in Copilot Studio states that test results remain available for 89 days and must be exported to CSV for longer storage. The documented test-case import uses CSV or TXT containing a question and expected answer.
An available import format does not establish that exports are complete. Check both directions and the information actually included.
What portability means here
Portability does not mean the same file runs unchanged in every tool. APIs, graders and trace formats differ; adapters may still be needed.
Outside the existing platform, your team should be able to explain:
- which behaviour was tested,
- which data and initial state applied,
- which outcomes counted as correct, incorrect or prohibited,
- how scoring was calculated,
- which model, prompt, tool and data versions were active,
- why the tested configuration was approved or rejected.
The practical evidence comes from applying the same business criteria in another test system. A readable archive is a starting point, not an execution test.
Eight parts of a portable evaluation package
This table is a suggested engineering checklist, not a formal standard.
| Asset | What to retain | Purpose |
|---|---|---|
| Test-case ID | Stable ID, title, category and provenance | Compare cases and outcomes across runs |
| Input and initial state | Request, role, process data and source references | Reconstruct the conditions applying to the case |
| Expected behaviour | Expected outcome, allowed variations and minimum requirements | Define business success |
| Prohibited behaviour | Critical failures, disallowed actions and escalation rules | Check boundaries even when the final answer is correct |
| Grader | Check, rubric, scale, threshold and reference data | Explain and recalculate scores |
| Run configuration | Model, prompt, tools, retrieval settings and versions | Trace changes |
| Case-level result | Output, check results, failure class, latency and relevant trace references | Identify individual critical failures |
| Release decision | Limits, exceptions, human review and rationale | Explain the operational decision |
RAG evaluations also need expected sources, document versions and relevant retrieval evidence. RAG evaluation with a golden dataset explains how to assess retrieval and answers separately.
For agents, tool calls, parameters, stopping conditions and expected state changes are also relevant. AI agent evaluation covers this execution layer.
Why two CSV columns may be insufficient
For a simple FAQ test, a question and expected answer may be enough. In a production workflow, the correct response can also depend on role, product group, document version or process state.
A reference answer alone does not automatically capture:
- required source citations,
- the valid response “information unavailable”,
- necessary clarifications,
- prohibited actions,
- permitted tool calls,
- the weight of a critical failure.
File format does not determine quality. CSV suits flat data; JSONL can represent nested state or multiple references. Document and version the schema either way.
A small, tool-independent package structure
| File or directory | Contents |
|---|---|
README.md | Purpose, scope, owner and execution instructions |
manifest.yaml | Package version, data classification and configuration |
cases.jsonl | Inputs, initial states and expectations |
rubrics/ | Scoring rubrics and label definitions |
checks/ | Deterministic checks |
runs/ | Case-level results from individual evaluation runs |
decisions/ | Approvals, deviations and justified exceptions |
This structure is a recommendation. A small assistant may need only a versioned table and understandable scoring rules.
The manifest should reference the workflow, package and schema versions, test-set version, rubric version and configuration versions. Also identify source data retained only by reference and explain how an authorised reviewer can restore it.
Graders must be as understandable as test cases
Deterministic checks
Rules such as “output satisfies the JSON schema”, “amount matches the business calculation” or “no write action occurred” should be retained as understandable code or declarative rules.
Store passing and failing examples too. The team can then check whether a reimplemented rule assesses the same behaviour.
Scoring rubrics
A rubric needs a criterion, defined scale, examples and threshold. Explicitly identify critical failures that cause rejection regardless of other subscores.
LLM-as-judge
If a model grades outputs, document its identifier, grader prompt, input fields, output format, parameters and calibration cases. Retain expert reference ratings for checking the grader.
OpenAI's evaluation best practices recommend calibrating automated scoring against human judgements and including typical, edge and adversarial cases.
A “quality score of 84” cannot be interpreted reliably without its rubric, grader version and case-level results. A different grader model may also score differently. Portability does not promise identical scores; it makes differences inspectable.
Fictional example: the download is insufficient for migration
A company evaluates a service assistant on 60 cases. Its dashboard shows 84% overall quality. The export includes every question, generated answer and the aggregate score.
It omits user roles, expected document versions, no-answer cases, the grader prompt, and model and prompt versions. This example is entirely fictional and does not describe a platform test.
The team can archive the visible content, but cannot explain how the 84% was calculated. It would have to reconstruct the evaluation basis before making a comparison.
A useful first step would be to restore a small subset outside the platform, including a role-sensitive case and a no-answer case. Missing information then shows what the regular export needs to include.
An exit test in five steps
- Export the package. Retain cases, rules, grader configurations, case-level results, annotations and versions. Check relationships as well as file counts.
- Select representative cases. Include a normal case, edge case, missing information, prohibited action and escalation where relevant to the workflow.
- Run outside the platform. Use a small alternative runner with the same business criteria. Execute tool and write actions only in controlled test environments.
- Classify differences. Separate missing data, non-portable graders and missing state from actual system-quality differences.
- Document effort. Record reusable components, required adapters, manual reconstruction and remaining gaps.
The result is an evidence-based estimate of exit effort. A successful download alone cannot provide it.
Questions before purchase or renewal
- Can test cases and stable IDs be exported in bulk in a machine-readable format?
- Are relationships to runs, case-level ratings and annotations preserved?
- Can custom rubrics, grader prompts and thresholds be exported?
- Are outputs, relevant traces and configuration versions included?
- How long are historical results retained?
- Is there a documented API export or only manual downloads?
- Are usage rights for test data, rubrics and derived results clear?
- Has restoration outside the platform actually been tested?
Technical portability and contractual usage rights are separate questions. A contractual export right does not establish that the technical export is useful.
Keep control of the exported data
Test cases and traces may contain confidential information. Retain the evidence needed for evaluation and protect exports according to their data classification. Credentials do not belong in the package; document required connections and permissions instead.
Export and deletion are separate operations. A successful archive says nothing about when copies in the previous platform are deleted. Check that platform's specific retention and deletion procedures.
Match the effort to the system
A small, fully human-reviewed assistant may need only an externally stored test set and clear rubrics. A business-critical agent with write access also needs traceable states, tool expectations and approvals.
More control brings maintenance responsibility. The architecture guide to private AI and public cloud helps allocate that responsibility deliberately.
A practical next step
Take an existing run and try to explain and rerun five different test cases using only the export. This is an initial gap check, not complete evidence for every supported task.
If you need to recover information from the provider interface or someone's memory, add the missing assets. For an existing system, this can form part of a focused AI Production Readiness & Reliability Sprint. We define the scope around your actual system.