Skip to main content

Replay datasets and runs

Section: DOC-IS-evaluation-datasets-and-runs#replay-datasets-and-runs.

Check that a prompt or model change still handles the examples your application depends on. Build a dataset of those examples with expected behavior or a stated scoring rubric, then use your test harness to run the baseline and candidate against the same set.

A run records the outcomes your evaluation harness submits. For reproducible comparisons, keep the versioned eligible set and failed or excluded cases in your harness manifest; a stored run does not guarantee that every item was evaluated.

This whole loop works with a bare API key — no user identity needed.

The loop​

Section: DOC-IS-evaluation-datasets-and-runs#the-loop.

Use a failed customer reply as a regression case before changing the prompt. You need evaluation enabled, permitted source input, a reviewed expectation and a test harness that can run both configurations. Keep real recipients and effectful tools out of the replay unless the test explicitly authorizes them.

The workflow below creates the dataset, curates its inputs and expectations, runs your harness, and compares the results. Current APIs do not automatically freeze full conversation/tool context or execute the dataset. Keep the harness manifest and item outcomes as the comparison record when platform run association is unavailable.

1. Create the dataset​

Section: DOC-IS-evaluation-datasets-and-runs#1-create-the-dataset.

curl -X POST https://api.travila.ai/api/v1/evals/create-dataset \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "name": "support-assistant-regressions",
  "description": "Turns we do not want to break"
}'

Reference: Create a dataset · Request fields.

Name it for what it protects. Runs are addressed by dataset name plus run name, so the name is a durable handle, not a label.

2. Save the input needed to reproduce the problem​

Section: DOC-IS-evaluation-datasets-and-runs#2-harvest-turns-into-it.

A turn address establishes source lineage. The current operation does not fetch the trace or conversation input; supply the intended inputJson explicitly, after applying your data policy:

curl -X POST https://api.travila.ai/api/v1/evals/add-dataset-item \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "datasetId": "support-assistant-regressions",
  "conversationId": "conv_123",
  "sourceUserMessageId": "msg_abc",
  "inputJson": "{\"question\":\"What is the return window?\"}"
}'

Reference: Add a dataset item · Request fields.

Read and curate the permitted source input explicitly. Keep its redaction, omissions and fixture provenance with the case; see input capture and replacement behavior.

3. Define what an acceptable answer must do​

Section: DOC-IS-evaluation-datasets-and-runs#3-author-the-expected-output.

Use an expert-reviewed answer, a deterministic validator or another qualified reference with recorded provenance. Automated construction can be useful, but a candidate model's own answer is not ground truth. Some evaluations use a rubric instead of an exact expected answer; specify which contract each item requires.

List cases that still need an expected output:

curl -X POST https://api.travila.ai/api/v1/evals/list-dataset-items \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "datasetName": "support-assistant-regressions",
  "onlyMissingExpectedOutput": true
}'

Reference: List dataset items · Request fields.

Use the identifier expected by each operation; the dataset identifier reference records the current differences.

Then fill each one in:

curl -X POST https://api.travila.ai/api/v1/evals/update-dataset-item \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "itemId": "…",
  "datasetId": "support-assistant-regressions",
  "expectedOutputJson": "{\"answer\": \"…\"}"
}'

Reference: Update a dataset item · Request fields.

Curate and count exclusions explicitly

The current record/replay operations do not automatically skip items with missing expectations. Your harness must enforce the evaluator's prerequisites and report failed, excluded and unevaluated counts separately. Never count an item with nothing scored as a pass, or silently reduce the denominator.

4. Run both configurations and retain their outcomes​

Section: DOC-IS-evaluation-datasets-and-runs#4-record-the-run.

Before recording platform associations:

  1. Freeze the cases and the inputs/tools/memory fixtures the harness will use.
  2. Run the baseline and candidate under the same relevant conditions and limits.
  3. Keep a result for every eligible case: evaluated, failed, excluded or not evaluated, with the reason.
  4. Preserve the configuration, rubric and item versions with those outcomes.
  5. Use the association call below only where supported, then read the expected run back.

This produces an experiment you can inspect even when the platform cannot associate every trace. A run name or success response is not a substitute for the harness’s actual results.

Read back the expected run before relying on the association. Keep the harness manifest and item outcomes when association is unavailable; see association limits.

curl -X POST https://api.travila.ai/api/v1/evals/record-dataset-run \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "datasetId": "support-assistant-regressions",
  "runName": "nightly-2026-08-14",
  "profileRevisionHash": "a1b2c3…",
  "items": [
    {
      "datasetItemId": "di_001",
      "traceId": "trc_a1b2c3"
    }
  ]
}'

Reference: Record a dataset run · Request fields.

Keep the harness’s versioned results and full configuration beside the platform record. The run contract distinguishes association metadata, supported score targets and reproducible inputs.

5. Decide whether the change fixes the problem without regressions​

Section: DOC-IS-evaluation-datasets-and-runs#5-compare.

Join baseline and candidate by the same versioned case, inspect changed outcomes, and review failures alongside score differences. Accept a change only against your stated criteria; keep inconclusive or missing results visible. Use the following reads to locate supported platform run records, then reconcile them with your harness report.

curl -X POST https://api.travila.ai/api/v1/evals/list-dataset-runs \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "datasetName": "support-assistant-regressions",
  "pageSize": 25
}'

Reference: List dataset runs · Request fields.

Response:

{
  "runs": [
    {
      "runId": "run_0912",
      "name": "2026-08-12-candidate",
      "datasetId": "ds_771",
      "description": "profile revision and qualified model recorded in manifest",
      "itemCount": 120,
      "createdAt": "2026-08-12T18:00:00Z"
    }
  ],
  "cursorPage": {
    "limit": 25
  }
}

Reference: List dataset runs · Response fields.

Continue through all result pages and compare them with the harness record. Use the pagination and enrichment contract when a count or expected run is missing.

Then pull one run by dataset name and run name — not by runId:

curl -X POST https://api.travila.ai/api/v1/evals/get-dataset-run \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "datasetName": "support-assistant-regressions",
  "runName": "2026-08-12-candidate"
}'

Reference: Get a dataset run · Request fields.

Response:

{
  "run": {
    "runId": "run_0912",
    "name": "2026-08-12-candidate",
    "datasetId": "ds_771",
    "itemCount": 120,
    "metadataJson": "{\"profileRevisionHash\":\"9f2c1e…\"}",
    "createdAt": "2026-08-12T18:00:00Z"
  }
}

Reference: Get a dataset run · Response fields.

Use the harness manifest to join baseline and candidate by the same versioned datasetItemId. get-dataset-run returns a run summary, not item-level results. Read scores through list-scores using verified run associations, and reconcile them with expected, failed and excluded cases; a returned run ID alone does not establish complete results.

Browse the datasets​

Section: DOC-IS-evaluation-datasets-and-runs#browse-the-datasets.

curl -X POST https://api.travila.ai/api/v1/evals/list-datasets \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "page": 1,
  "pageSize": 25
}'

Reference: List datasets · Request fields.

Response:

{
  "datasets": [
    {
      "datasetId": "ds_771",
      "name": "support-assistant-regressions",
      "description": "Turns a reviewer marked as wrong",
      "itemCount": 120,
      "runCount": 8,
      "lastRunAt": "2026-08-12T18:00:00Z",
      "createdAt": "2026-06-02T10:00:00Z"
    }
  ],
  "page": {
    "page": 1,
    "limit": 25,
    "totalItems": 3,
    "totalPages": 1
  }
}

Reference: List datasets · Response fields.

Confirm execution from the expected run and harness record; history enrichment limits can leave summary counts incomplete.

Maintain the regression set as the application changes​

Section: DOC-IS-evaluation-datasets-and-runs#keeping-a-dataset-honest.

Two failure modes are worth designing against, because neither announces itself:

Frozen context ages. Explicitly captured inputs and fixtures preserve selected context from a turn. The current API does not automatically capture memory, tools or prompt variables as a frozen fixture. A complete versioned manifest helps make runs comparable, and it is also what makes a two-year-old golden set measure a world that no longer exists. Re-harvest from recent traffic periodically and version the dataset name so a run declares which vintage it measured.

Control the comparison. Hold unrelated variables and the dataset version fixed when isolating one cause. If you intentionally test a combined change, label it as such. Report sample sizes, missing outcomes and uncertainty; a score difference alone does not establish causation.

Upcoming: experiment context at message ingress​

Status: Upcoming — not yet available.

Section: DOC-IS-evaluation-datasets-and-runs#experiment-context-at-ingress

Start an evaluation message with its dataset, run and item identified. Its recorded observations stay associated with that experiment, including when generation waits on tools, so you do not need a later association call.

This reference does not yet document that request input. Until the customer interface is available, keep your harness manifest and inspect each expected run/item association. A successful model call or an empty association response does not confirm that the experiment was recorded.

Example: Your nightly regression harness submits each case with its experiment context. A turn that pauses for a tool keeps that case association when it resumes. The public request fields for this input are not yet available.

Document ID: DOC-IS-evaluation-datasets-and-runs. Section identities and revisions.