Skip to main content

Replay datasets and runs

Section ID: Guide/jf05ium0qn#replay-datasets-and-runs

Check that a prompt or model change still handles the examples your application depends on. Build a dataset of those examples with expected behavior or a stated scoring rubric, then use your test harness to run the baseline and candidate against the same set.

A run records the outcomes your evaluation harness submits. For reproducible comparisons, keep the versioned eligible set and failed or excluded cases in your harness manifest; a stored run does not guarantee that every item was evaluated.

Dataset and run management accepts a public API key. Replaying cases through a conversation also needs the authorized user identity required by the message operation. Recording human scores needs a verified reviewer identity; see review and scoring.

The loop​

Section ID: Guide/jf05ium0qn#the-loop

Use a failed customer reply as a regression case before changing the prompt. You need evaluation enabled, permitted source input, a reviewed expectation and a test harness that can run both configurations. Keep real recipients and effectful tools out of the replay unless the test explicitly authorizes them.

The workflow below creates the dataset, curates its inputs and expectations, runs your harness, and compares the results. Current APIs do not automatically freeze full conversation/tool context or execute the dataset. Keep the harness manifest and item outcomes as the comparison record when platform run association is unavailable.

Dataset replay and comparison loop

An application creates a dataset, curates inputs and expectations, runs its harness, records the experiment and reads stored run results.

1. Create the dataset​

Section ID: Guide/jf05ium0qn#1-create-the-dataset

Request example: Create a dataset. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Name it for what it protects. Runs are addressed by dataset name plus run name, so the name is a durable handle, not a label.

2. Save the input needed to reproduce the problem​

Section ID: Guide/jf05ium0qn#2-harvest-turns-into-it

A turn address establishes source lineage. The current operation does not fetch the trace or conversation input; supply the intended inputJson explicitly, after applying your data policy:

Request example: Add a dataset item. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Read and curate the permitted source input explicitly. Keep its redaction, omissions and fixture provenance with the case; see input capture and replacement behavior.

3. Define what an acceptable answer must do​

Section ID: Guide/jf05ium0qn#3-author-the-expected-output

Use an expert-reviewed answer, a deterministic validator or another qualified reference with recorded provenance. Automated construction can be useful, but a candidate model's own answer is not ground truth. Some evaluations use a rubric instead of an exact expected answer; specify which contract each item requires.

List cases that still need an expected output:

Request example: List dataset items. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Use the identifier expected by each operation; the dataset identifier reference records the current differences.

Then fill each one in:

Request example: Update a dataset item. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Warning: Curate and count exclusions explicitly

The current record/replay operations do not automatically skip items with missing expectations. Your harness must enforce the evaluator's prerequisites and report failed, excluded and unevaluated counts separately. Never count an item with nothing scored as a pass, or silently reduce the denominator.

4. Run both configurations and retain their outcomes​

Section ID: Guide/jf05ium0qn#4-record-the-run

Before sending the evaluation turns:

  1. Freeze the cases and the inputs/tools/memory fixtures the harness will use.

  2. Run the baseline and candidate under the same relevant conditions and limits.

  3. Keep a result for every eligible case: evaluated, failed, excluded or not evaluated, with the reason.

  4. Preserve the configuration, rubric and item versions with those outcomes.

  5. Attribute each turn to its run when you send it, then read the expected run back.

A turn joins a dataset run only through the experiment block on the message that starts it — send-message-sync or send-message. There is no call that associates a trace with a run after it has been recorded. Send one turn per case, each carrying the dataset, the run name and the case's dataset item:

Request example: Send a message and wait for the result. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; X-On-Behalf-Of: user_123; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

  • datasetId is the dataset's ID as returned by create-dataset or list-datasets, not its name.

  • runName names the run. Reusing a run name for the same dataset adds turns to that run.

  • datasetItemId identifies the case this turn exercises. Set it on every turn of a run: turns sent without it currently land in a different run from turns that carry it, even under the same run name.

  • description, metadataJson and profileRevisionHash are optional run metadata; see association limits.

With send-message, the call returns before the turn finishes; wait for the turn to complete before reading the run back or scoring it. A run name or success response is not a substitute for the harness’s actual results: read the expected run back with list-dataset-runs, which takes the dataset name, and keep the harness manifest and item outcomes when an expected run or item is missing.

5. Decide whether the change fixes the problem without regressions​

Section ID: Guide/jf05ium0qn#5-compare

Join baseline and candidate by the same versioned case, inspect changed outcomes, and review failures alongside score differences. Accept a change only against your stated criteria; keep inconclusive or missing results visible. Use the following reads to locate supported platform run records, then reconcile them with your harness report.

Request example: List dataset runs. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Response:

Response example: List dataset runs. Example identifiers and values are illustrative; substitute authorized values.

Continue through all result pages and compare them with the harness record. Use the pagination and enrichment contract when a count or expected run is missing.

Then pull one run by dataset name and run name — not by runId:

Request example: Get a dataset run. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Response:

Response example: Get a dataset run. Example identifiers and values are illustrative; substitute authorized values.

Use the harness manifest to join baseline and candidate by the same versioned datasetItemId. get-dataset-run returns a run summary, not item-level results. Read scores through list-scores using verified run associations, and reconcile them with expected, failed and excluded cases; a returned run ID alone does not establish complete results.

Browse the datasets​

Section ID: Guide/jf05ium0qn#browse-the-datasets

Request example: List datasets. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Response:

Response example: List datasets. Example identifiers and values are illustrative; substitute authorized values.

Confirm execution from the expected run and harness record; history enrichment limits can leave summary counts incomplete.

Maintain the regression set as the application changes​

Section ID: Guide/jf05ium0qn#keeping-a-dataset-honest

Two failure modes are worth designing against, because neither announces itself:

Frozen context ages. Explicitly captured inputs and fixtures preserve selected context from a turn. The current API does not automatically capture memory, tools or prompt variables as a frozen fixture. A complete versioned manifest helps make runs comparable, and it is also what makes a two-year-old golden set measure a world that no longer exists. Re-harvest from recent traffic periodically and version the dataset name so a run declares which vintage it measured.

Control the comparison. Hold unrelated variables and the dataset version fixed when isolating one cause. If you intentionally test a combined change, label it as such. Report sample sizes, missing outcomes and uncertainty; a score difference alone does not establish causation.

Experiment context at message ingress​

Section ID: Guide/jf05ium0qn#experiment-context-at-ingress

A turn is attached to a dataset run when you send it: set the experiment block on send-message-sync or send-message. Step 4 shows the request.

  • datasetId and runName are required. Use the dataset ID returned by create-dataset or list-datasets, not the dataset name.

  • There is no later association call; a turn sent without the block is not part of any run.

  • A successful send does not confirm the run was recorded. Read the run back with list-dataset-runs and keep your harness manifest.

Document ID: Guide/jf05ium0qn