Skip to main content

Review and score turns

Section: DOC-IS-evaluation-review#review-and-score-turns.

Review replies that need a person’s judgment, such as an answer that is accurate but hard to use. Give reviewers a common rating scale, queue the turns and record both scores and explanations. Use the reviewed examples to improve regression datasets and calibrate automated judges.

This loop needs a user identity

record-score rejects a bare API key because the score must be attributable to a reviewer. Use an authorized public API key with verified X-On-Behalf-Of; console JWTs belong to the separate console surface. Score configs, queues and comments accept a bare key. See record-score for the exact contract.

The loop​

Section: DOC-IS-evaluation-review#the-loop.

Build a weekly review queue for coaching replies whose helpfulness needs a person’s judgment. You need evaluation enabled, a backend credential, a verified reviewer identity for scoring, and trace IDs from the investigation recipe.

  1. Define a shared helpfulness scale and an explanation of what each rating means.
  2. Create a queue using that score configuration and add the selected traces.
  3. Let the reviewer read a queued trace, record the score and explain it.
  4. Complete the queue item only after the review is saved.
  5. Turn a useful failure into a curated regression case.

The result is a completed work item plus an attributable judgment. Completing the queue is a separate operation from saving the score; neither guarantees that the judgment is correct.

create-score-config ──► create-annotation-queue ──► enqueue-for-annotation
   (the dimensions)        (bound to those configs)      (what to review)
                                                                │
                                                                ▼
       complete-annotation-queue-item ◄── record-score ◄── list-annotation-queue-items
                                          create-comment

Keep reviewer judgments separate from user sentiment​

Section: DOC-IS-evaluation-review#never-blend-score-sources.

Keep customer sentiment, expert review and automated measurements separate. Choose the score source for the question you are answering before computing an aggregate; use reviewer scoring for this workflow.

1. Define the dimensions first​

Section: DOC-IS-evaluation-review#1-define-the-dimensions-first.

A score config is a rating scale. Reviewers pick from it rather than inventing their own, which is what makes two reviewers' labels comparable.

curl -X POST https://api.travila.ai/api/v1/evals/create-score-config \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "name": "helpfulness",
  "dataType": "SCORE_DATA_TYPE_NUMERIC",
  "minValue": 1,
  "maxValue": 5
}'

Reference: Create a score config · Request fields.

Create and save a numeric or boolean trace-scoring configuration before recording ratings. Use the supported write and configuration contract when designing a different scale.

2. Queue what needs reviewing​

Section: DOC-IS-evaluation-review#2-queue-what-needs-reviewing.

Save the returned configuration ID and replace cfg_helpfulness below with it. A queue is a worklist bound to a fixed set of configs, so everyone reviewing from it scores the same dimensions. create-annotation-queue requires at least one entry in scoreConfigIds — a queue with no configs is rejected.

curl -X POST https://api.travila.ai/api/v1/evals/create-annotation-queue \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "name": "weekly-qa",
  "scoreConfigIds": [
    "cfg_helpfulness"
  ]
}'

Reference: Create an annotation queue · Request fields.

Save the returned queue ID and replace q_weekly in later requests. Enqueue the trace you want reviewed, using EVAL_TARGET_TYPE_TRACE for the currently qualified score path. The target-type contract lists other target kinds; their presence in the enum does not qualify them for this scoring recipe.

curl -X POST https://api.travila.ai/api/v1/evals/enqueue-for-annotation \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "queueId": "q_weekly",
  "targetType": "EVAL_TARGET_TYPE_TRACE",
  "targetId": "trc_a1b2c3"
}'

Reference: Enqueue an item for annotation · Request fields.

Pull the worklist​

Section: DOC-IS-evaluation-review#pull-the-worklist.

curl -X POST https://api.travila.ai/api/v1/evals/list-annotation-queue-items \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{
  "queueId": "q_weekly",
  "status": "ANNOTATION_QUEUE_ITEM_STATUS_PENDING",
  "page": 1,
  "pageSize": 20
}'

Reference: List annotation queue items · Request fields.

Response:

{
  "items": [
    {
      "itemId": "qi_001",
      "queueId": "q_weekly",
      "objectType": "EVAL_TARGET_TYPE_TRACE",
      "objectId": "trc_a1b2c3",
      "status": "ANNOTATION_QUEUE_ITEM_STATUS_PENDING",
      "createdAt": "2026-08-12T17:00:00Z"
    }
  ],
  "page": {
    "page": 1,
    "limit": 20,
    "totalItems": 34,
    "totalPages": 2
  }
}

Reference: List annotation queue items · Response fields.

Open the queued trace and retain separate evidence when you need durable completion attribution; see queue addressing and completion attribution.

3. Record the rating​

Section: DOC-IS-evaluation-review#3-record-the-rating.

Read the trace and apply the configured rubric before submitting the score. Use the reviewer’s verified identity and the saved score-config ID. After saving, read the stored score; serialize repeat actions so a delayed request cannot silently become a second review.

curl -X POST https://api.travila.ai/api/v1/evals/record-score \
  -H "X-API-Key: sk_your_key_here" \
  -H "X-On-Behalf-Of: reviewer@yourcompany.com" \
  -H "Content-Type: application/json" \
  -d '{
  "targetType": "EVAL_TARGET_TYPE_TRACE",
  "targetId": "…",
  "name": "helpfulness",
  "dataType": "SCORE_DATA_TYPE_NUMERIC",
  "numericValue": 4,
  "configId": "cfg_helpfulness",
  "comment": "Accurate, but buried the answer in three paragraphs."
}'

Reference: Record a score · Request fields.

Reconcile the stored score after repeated actions; see reviewer identity and replacement limits.

Save the explanation with the reviewed turn​

Section: DOC-IS-evaluation-review#comments.

Explain why the reply received its rating, such as a tool returning stale data. Attach the comment to the same reviewed trace and supply the reviewer’s identity so another person can understand who made the observation:

curl -X POST https://api.travila.ai/api/v1/evals/create-comment \
  -H "X-API-Key: sk_your_key_here" -H "X-On-Behalf-Of: reviewer@yourcompany.com" \
  -H "Content-Type: application/json" \
  -d '{
  "objectType": "COMMENT_OBJECT_TYPE_TRACE",
  "objectId": "trc_a1b2c3",
  "content": "Tool call returned stale data."
}'

Reference: Create a comment · Request fields.

Keep the returned comment ID, then read the trace’s comments to confirm the explanation was saved before completing the review. Keep the note relevant and under the 3000-character limit; longer content is rejected. A bare-key comment has no author.

The comment reference defines other supported targets. For this queue, continue using the trace you scored. If an authorized administrator needs to remove a note, delete-comment suppresses it from the list while retaining the record for audit. It does not erase every retained copy.

4. Complete the saved review​

Section: DOC-IS-evaluation-review#mark-it-done.

Call this after recording the rating and any explanatory comment, not when the reviewer merely opens the task. If recording failed, leave the item pending while you recover the score. Keep separate evidence if you need durable completion-author attribution: the current completion operation does not retain it.

curl -X POST https://api.travila.ai/api/v1/evals/complete-annotation-queue-item \
  -H "X-API-Key: sk_your_key_here" -H "X-On-Behalf-Of: reviewer_9" \
  -H "Content-Type: application/json" \
  -d '{
  "queueId": "q_weekly",
  "itemId": "qi_001"
}'

Reference: Complete an annotation queue item · Request fields.

Response:

{
  "item": {
    "itemId": "qi_001",
    "queueId": "q_weekly",
    "status": "ANNOTATION_QUEUE_ITEM_STATUS_COMPLETED",
    "completedAt": "2026-08-12T17:14:22Z",
    "completedBy": "reviewer_9"
  }
}

Reference: Complete an annotation queue item · Response fields.

Completing is what stops an item being handed out again — it is not implied by recording a score. A reviewer who scores but never completes leaves the item in the pending pool for the next person.

Closing the loop​

Section: DOC-IS-evaluation-review#closing-the-loop.

A reviewer who finds a bad turn should not stop at scoring it — add it to a dataset so it becomes an explicit regression case. Supply the input and reviewed expectation as described in dataset curation; the turn address alone does not capture the context.

Human review can provide calibration examples for an automated judge. Use a clear rubric, resolve reviewer disagreements and evaluate the judge on separate examples. Completing a queue alone does not establish that its labels are correct or that the judge will be reliable on other cases.

Choose a different rating scale​

Section: DOC-IS-evaluation-review#score-config-types-in-depth.

The recipe uses a 1–5 helpfulness rating. Use boolean trace scoring when a simple pass/fail judgment better matches the criterion. The schema also names categorical/text and other target types, but the current score write path supports numeric/boolean values on traces only.

Numeric bounds are UI hints, not enforced score limits; validate the reviewer’s choice in your application. The data type and categorical values cannot change after creation. For a different scale, create a new configuration so earlier scores retain their meaning. See the score-configuration reference for exact settings.

Retire a rubric without reinterpreting earlier scores​

Section: DOC-IS-evaluation-review#list-and-retire-configs.

When the helpfulness rubric changes, list its configurations, archive the old one through update-score-config, and create a replacement. Configurations are archived rather than deleted so earlier scores keep their reference.

Include archived configurations when displaying historical reviews. Changing a used configuration’s numeric bounds does not rescale its earlier scores; prefer a replacement when the meaning of the scale changes. Use the returned new configuration ID for future queues and scores.

Return to an existing review queue​

Section: DOC-IS-evaluation-review#find-the-queues.

Save the queue ID created for weekly review. When reviewers return, use that identity to load its pending items rather than creating another queue.

If the application needs a queue picker, use list-annotation-queues and its pagination contract. Confirm the chosen queue’s rubric before presenting its work; a similarly named queue can use a different scoring configuration.

Withdraw a rating​

Section: DOC-IS-evaluation-review#withdraw-a-rating.

curl -X POST https://api.travila.ai/api/v1/evals/delete-score \
  -H "X-API-Key: sk_your_key_here" \
  -H "X-On-Behalf-Of: reviewer@yourcompany.com" \
  -H "Content-Type: application/json" \
  -d '{
  "scoreId": "scr_7788"
}'

Reference: Delete a score · Request fields.

Response:

{
  "deleted": true
}

Reference: Delete a score · Response fields.

Use trusted administrative tooling for this operation and read back the result, allowing for delayed views. The score deletion contract explains the current authorization limit.

Document ID: DOC-IS-evaluation-review. Section identities and revisions.