Review

Human review answers a different question from a dataset run: not did anything regress, but was this specific reply any good, and why. It produces the labels that datasets and automated judges are calibrated against.

This loop needs a user identity

record-score rejects a bare API key — see Authentication. Use a console JWT or X-On-Behalf-Of. Score configs, queues and comments accept a bare key.

The loop

create-score-config ──► create-annotation-queue ──► enqueue-for-annotation
   (the dimensions)        (bound to those configs)      (what to review)
                                                                │
                                                                ▼
       complete-annotation-queue-item ◄── record-score ◄── list-annotation-queue-items
                                          create-comment

1. Define the dimensions first

A score config is a rating scale. Reviewers pick from it rather than inventing their own, which is what makes two reviewers' labels comparable.

curl -X POST https://api.travila.ai/api/v1/evals/create-score-config \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"name": "helpfulness", "dataType": "SCORE_DATA_TYPE_NUMERIC", "minValue": 1, "maxValue": 5}'

Four types are available: NUMERIC with an optional range, CATEGORICAL with labelled values, BOOLEAN, and TEXT.

Create the config before the score — configId is effectively required

record-score writes a human-source score, and a human-source score is rejected without a config id. The schema does not mark configId required, but in practice every call needs one:

{"code": 400, "message": "config_id is required for a HUMAN-source score: it maps to Langfuse's ANNOTATION source, which is rejected without one. Seed a score config for \"helpfulness\" first"}

Configs are archived rather than deleted (update-score-config with isArchived), because deleting one would orphan every score recorded against it. list-score-configs takes includeArchived when you need the retired ones.

List and retire configs

curl -X POST https://api.travila.ai/api/v1/evals/list-score-configs \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"page": 1, "pageSize": 50}'

Response:

{
  "configs": [
    {
      "configId": "cfg_helpfulness",
      "name": "helpfulness",
      "dataType": "SCORE_DATA_TYPE_NUMERIC",
      "minValue": 1,
      "maxValue": 5,
      "createdAt": "2026-06-02T10:00:00Z"
    }
  ],
  "page": {"page": 1, "limit": 50, "totalItems": 1, "totalPages": 1}
}

Archived configs are excluded unless you send includeArchived: true — which you need when rendering historical scores, since those still reference the retired config.

There is no delete. Retiring a dimension is an update:

curl -X POST https://api.travila.ai/api/v1/evals/update-score-config \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
        "configId": "cfg_tone",
        "name": "tone",
        "isArchived": true
      }'

Response: {"config": {...}} with isArchived: true.

name is required even when you are only flipping the archive flag. Changing minValue or maxValue on a config that already has scores against it does not rescale them — the old scores keep their raw values on the new scale, so prefer archiving and creating a replacement.

2. Queue what needs reviewing

A queue is a worklist bound to a fixed set of configs, so everyone reviewing from it scores the same dimensions.

curl -X POST https://api.travila.ai/api/v1/evals/create-annotation-queue \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"name": "weekly-qa", "scoreConfigIds": ["cfg_helpfulness", "cfg_tone"]}'

Then put work in it. targetType is one of EVAL_TARGET_TYPE_TRACE, EVAL_TARGET_TYPE_OBSERVATION, EVAL_TARGET_TYPE_SESSION or EVAL_TARGET_TYPE_DATASET_RUN:

curl -X POST https://api.travila.ai/api/v1/evals/enqueue-for-annotation \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"queueId": "q_weekly", "targetType": "EVAL_TARGET_TYPE_TRACE", "targetId": "trc_a1b2c3"}'

Find the queues

curl -X POST https://api.travila.ai/api/v1/evals/list-annotation-queues \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"page": 1, "pageSize": 25}'

Response:

{
  "queues": [
    {
      "queueId": "q_weekly",
      "name": "weekly-qa",
      "description": "Sampled turns from the coaching profiles",
      "scoreConfigIds": ["cfg_helpfulness", "cfg_tone"],
      "pendingCount": 34,
      "completedCount": 112,
      "createdAt": "2026-06-02T10:00:00Z"
    }
  ],
  "page": {"page": 1, "limit": 25, "totalItems": 1, "totalPages": 1}
}

pendingCount and completedCount are what a reviewer dashboard renders as progress — no need to page the items to count them.

Pull the worklist

curl -X POST https://api.travila.ai/api/v1/evals/list-annotation-queue-items \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
        "queueId": "q_weekly",
        "status": "ANNOTATION_QUEUE_ITEM_STATUS_PENDING",
        "page": 1,
        "pageSize": 20
      }'

Response:

{
  "items": [
    {
      "itemId": "qi_001",
      "queueId": "q_weekly",
      "objectType": "EVAL_TARGET_TYPE_TRACE",
      "objectId": "trc_a1b2c3",
      "status": "ANNOTATION_QUEUE_ITEM_STATUS_PENDING",
      "createdAt": "2026-08-12T17:00:00Z"
    }
  ],
  "page": {"page": 1, "limit": 20, "totalItems": 34, "totalPages": 2}
}

Feed objectId into get-trace to render what the reviewer is judging. Omit status to see completed items too — they carry completedAt and completedBy.

Mark it done

curl -X POST https://api.travila.ai/api/v1/evals/complete-annotation-queue-item \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"queueId": "q_weekly", "itemId": "qi_001"}'

Response:

{
  "item": {
    "itemId": "qi_001",
    "queueId": "q_weekly",
    "status": "ANNOTATION_QUEUE_ITEM_STATUS_COMPLETED",
    "completedAt": "2026-08-12T17:14:22Z",
    "completedBy": "reviewer_9"
  }
}

Completing is what stops an item being handed out again — it is not implied by recording a score. A reviewer who scores but never completes leaves the item in the pending pool for the next person.

3. Record the rating

curl -X POST https://api.travila.ai/api/v1/evals/record-score \
  -H "X-API-Key: sk_your_key_here" \
  -H "X-On-Behalf-Of: reviewer@yourcompany.com" \
  -H "Content-Type: application/json" \
  -d '{
        "targetType": "EVAL_TARGET_TYPE_TRACE",
        "targetId": "…",
        "name": "helpfulness",
        "dataType": "SCORE_DATA_TYPE_NUMERIC",
        "numericValue": 4,
        "configId": "cfg_helpfulness",
        "comment": "Accurate, but buried the answer in three paragraphs."
      }'

Two properties are worth relying on:

• Re-rating replaces, it does not duplicate. The score id is derived per rater and target, so posting again updates in place. There is no duplicate-per-click failure mode, and one reviewer changing their mind never touches another's score.
• The rater is server-stamped. It comes from your verified identity, never from the body. There is no field for it, which is why nobody can write or clear a colleague's rating.

Withdraw a rating

curl -X POST https://api.travila.ai/api/v1/evals/delete-score \
  -H "X-API-Key: sk_your_key_here" \
  -H "X-On-Behalf-Of: reviewer@yourcompany.com" \
  -H "Content-Type: application/json" \
  -d '{"scoreId": "scr_7788"}'

Response:

{
  "deleted": true
}

It withdraws your own rating; withdrawing one that was never there is a success rather than an error, and returns deleted: false — which, being false, is omitted from the response entirely. Get the scoreId from list-scores.

Never blend score sources

Scores carry a source, and mixing them produces a number that means nothing:

Source	Written by	Means
SCORE_SOURCE_USER	The end-user rating path	An end user said this. Read-only here.
SCORE_SOURCE_HUMAN	This API	A reviewer on your team said this.
SCORE_SOURCE_JUDGE / SCORE_SOURCE_EVAL	Automated evaluators	A model said this. Read-only.
SCORE_SOURCE_HARNESS	A dataset run	A harness measured this.

An average across sources is a misleading number

A 1-5 from an end user and a 1-5 from a domain reviewer are different measurements. Blending staff opinion into end-user sentiment corrupts the signal silently and irreversibly — and end-user sentiment is what feeds back into the product.

Always filter list-scores by source when aggregating. This is also why reviewer ratings are recorded here and never through the message-rating API: an end user's rating changes what the agent says to them next, and a review must have no side effect on the thing under review.

Comments

Comments attach a note to a trace, observation or session, so context travels with the artifact instead of in a side channel:

curl -X POST https://api.travila.ai/api/v1/evals/create-comment \
  -H "X-API-Key: sk_your_key_here" -H "X-On-Behalf-Of: reviewer@yourcompany.com" \
  -H "Content-Type: application/json" \
  -d '{
        "objectType": "COMMENT_OBJECT_TYPE_TRACE",
        "objectId": "trc_a1b2c3",
        "content": "Tool call returned stale data."
      }'

Response: {"commentId": "cmt_5501"}.

objectType is one of COMMENT_OBJECT_TYPE_TRACE, COMMENT_OBJECT_TYPE_OBSERVATION or COMMENT_OBJECT_TYPE_SESSION — the full enum name, not the bare word.

A comment sent with a bare API key is stored without an author. Send an identity if you want to know who said it.

Read them back for one object, or across the project:

curl -X POST https://api.travila.ai/api/v1/evals/list-comments \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
        "objectType": "COMMENT_OBJECT_TYPE_TRACE",
        "objectId": "trc_a1b2c3",
        "page": 1,
        "pageSize": 25
      }'

Response:

{
  "comments": [
    {
      "commentId": "cmt_5501",
      "objectType": "COMMENT_OBJECT_TYPE_TRACE",
      "objectId": "trc_a1b2c3",
      "content": "Tool call returned stale data.",
      "authorUserId": "reviewer_9",
      "createdAt": "2026-08-12T17:20:00Z"
    }
  ],
  "page": {"page": 1, "limit": 25, "totalItems": 1, "totalPages": 1}
}

Omit objectId and objectType to list every comment in the project — useful for a "what did reviewers say this week" digest.

curl -X POST https://api.travila.ai/api/v1/evals/delete-comment \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"commentId": "cmt_5501"}'

Response: {"deleted": true}.

Closing the loop

A reviewer who finds a bad turn should not stop at scoring it — add it to a dataset so it stays fixed. You already have the turn address, which is all add-dataset-item needs.

Human labels also have a second life: they are the ground truth an automated judge is calibrated against. A queue completed by people is what tells you whether a model-based judge can be trusted to score the rest.