Skip to main content

Datasets and runs

A dataset is a golden set: turns you care about not regressing, each paired with the answer you wish the agent had given. A run is one execution of a harness over that dataset, with a score per item.

This whole loop works with a bare API key — no user identity needed.

The loop​

create-dataset ──► add-dataset-item ──► update-dataset-item ──► (your harness runs)
                   (harvest the turn)    (author the expectation)          │
                                                                           ▼
              get-dataset-run ◄── list-dataset-runs ◄────────── record-dataset-run

1. Create the dataset​

curl -X POST https://api.travila.ai/api/v1/evals/create-dataset \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"name": "nutrition-coach-regressions", "description": "Turns we do not want to break"}'

Name it for what it protects. Runs are addressed by dataset name plus run name, so the name is a durable handle, not a label.

2. Harvest turns into it​

You do not need a trace id. Add a turn by its address:

curl -X POST https://api.travila.ai/api/v1/evals/add-dataset-item \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
        "datasetId": "nutrition-coach-regressions",
        "conversationId": "conv_123",
        "sourceUserMessageId": "msg_abc"
      }'
Harvesting the same turn twice yields one item

The item id is derived server-side from (dataset, turn). There is no way to supply your own, so a curator clicking twice or a redelivered event cannot create a duplicate.

Because the key includes the dataset, the same turn added to two different datasets is correctly two items — a regression set and a golden set are different things.

Leave inputJson empty and the service lifts the input from the trace. Remember that trace content is redacted, so an item harvested this way captures redacted context.

3. Author the expected output​

This is the step that cannot be automated. A golden answer has no source anywhere in the platform — it is authored, not observed. So harvesting deliberately creates items with a null expectation, parked for a human to complete.

Find them:

curl -X POST https://api.travila.ai/api/v1/evals/list-dataset-items \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"datasetId": "nutrition-coach-regressions", "onlyMissingExpectedOutput": true}'

Then fill each one in:

curl -X POST https://api.travila.ai/api/v1/evals/update-dataset-item \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
        "itemId": "…",
        "datasetId": "nutrition-coach-regressions",
        "expectedOutputJson": "{\"answer\": \"…\"}"
      }'
An item with no expectation is excluded from a run, not counted as a pass

Scoring nothing is not scoring a success. Runs skip these items entirely — which means a dataset that is half-curated silently measures half of what you think it does. Check onlyMissingExpectedOutput before trusting a run's numbers.

4. Record the run​

Your harness executes the dataset however it likes — this API does not run anything. When it has results, post the whole run in one call:

curl -X POST https://api.travila.ai/api/v1/evals/record-dataset-run \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
        "datasetId": "nutrition-coach-regressions",
        "runName": "nightly-2026-08-14",
        "profileRevisionHash": "a1b2c3…",
        "items": [
          {"datasetItemId": "…", "traceId": "…", "score": 1.0}
        ]
      }'
profileRevisionHash is what makes two runs comparable

It pins the run to the exact agent-profile revision under test. Without it you can see that a number moved but not what moved it — and comparing runs across an unrecorded prompt change is how a regression gets attributed to the wrong cause.

It is the same hash that appears as configHash on traces, so a run and the live traffic it predicts can be joined on it.

5. Compare​

curl -X POST https://api.travila.ai/api/v1/evals/list-dataset-runs \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"datasetName": "nutrition-coach-regressions"}'

get-dataset-run takes the dataset name and run name and returns the per-item results, so a candidate run can be diffed against the baseline it is trying to beat.

Keeping a dataset honest​

Two failure modes are worth designing against, because neither announces itself:

Frozen context ages. An item captures the turn's context as it was — the memories, tool outputs and prompt variables of that moment. That freeze is what makes runs comparable, and it is also what makes a two-year-old golden set measure a world that no longer exists. Re-harvest from recent traffic periodically and version the dataset name so a run declares which vintage it measured.

One variable per run. Change the profile revision or the model, not both. Everything else should replay from the frozen envelope, or the score delta has no single cause.