Review

Human review answers a different question from a dataset run: not did anything regress, but was this specific reply any good, and why. It produces the labels that datasets and automated judges are calibrated against.

This loop needs a user identity

record-score rejects a bare API key — see Authentication. Use a console JWT or X-On-Behalf-Of. Score configs, queues and comments accept a bare key.

The loop

create-score-config ──► create-annotation-queue ──► enqueue-for-annotation
   (the dimensions)        (bound to those configs)      (what to review)
                                                                │
                                                                ▼
       complete-annotation-queue-item ◄── record-score ◄── list-annotation-queue-items
                                          create-comment

1. Define the dimensions first

A score config is a rating scale. Reviewers pick from it rather than inventing their own, which is what makes two reviewers' labels comparable.

curl -X POST https://api.travila.ai/api/v1/evals/create-score-config \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"name": "helpfulness", "dataType": "SCORE_DATA_TYPE_NUMERIC", "minValue": 1, "maxValue": 5}'

Four types are available: NUMERIC with an optional range, CATEGORICAL with labelled values, BOOLEAN, and TEXT.

Create the config before the score — configId is effectively required

record-score writes a human-source score, and a human-source score is rejected without a config id. The schema does not mark configId required, but in practice every call needs one:

{"code": 400, "message": "config_id is required for a HUMAN-source score: it maps to Langfuse's ANNOTATION source, which is rejected without one. Seed a score config for \"helpfulness\" first"}

Configs are archived rather than deleted (update-score-config with isArchived), because deleting one would orphan every score recorded against it. list-score-configs takes includeArchived when you need the retired ones.

2. Queue what needs reviewing

A queue is a worklist bound to a fixed set of configs, so everyone reviewing from it scores the same dimensions.

curl -X POST https://api.travila.ai/api/v1/evals/create-annotation-queue \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"name": "weekly-qa", "scoreConfigIds": ["cfg_helpfulness", "cfg_tone"]}'

curl -X POST https://api.travila.ai/api/v1/evals/enqueue-for-annotation \
  -H "X-API-Key: sk_your_key_here" -H "Content-Type: application/json" \
  -d '{"queueId": "q_weekly", "targetType": "EVAL_TARGET_TYPE_TRACE", "targetId": "…"}'

Reviewers pull pending work with list-annotation-queue-items filtered on status, and mark each one done with complete-annotation-queue-item so it stops being handed out.

3. Record the rating

curl -X POST https://api.travila.ai/api/v1/evals/record-score \
  -H "X-API-Key: sk_your_key_here" \
  -H "X-On-Behalf-Of: reviewer@yourcompany.com" \
  -H "Content-Type: application/json" \
  -d '{
        "targetType": "EVAL_TARGET_TYPE_TRACE",
        "targetId": "…",
        "name": "helpfulness",
        "dataType": "SCORE_DATA_TYPE_NUMERIC",
        "numericValue": 4,
        "configId": "cfg_helpfulness",
        "comment": "Accurate, but buried the answer in three paragraphs."
      }'

Two properties are worth relying on:

• Re-rating replaces, it does not duplicate. The score id is derived per rater and target, so posting again updates in place. There is no duplicate-per-click failure mode, and one reviewer changing their mind never touches another's score.
• The rater is server-stamped. It comes from your verified identity, never from the body. There is no field for it, which is why nobody can write or clear a colleague's rating.

delete-score withdraws your own rating, and withdrawing one that was never there is a success rather than an error.

Never blend score sources

Scores carry a source, and mixing them produces a number that means nothing:

Source	Written by	Means
SCORE_SOURCE_USER	The end-user rating path	An end user said this. Read-only here.
SCORE_SOURCE_HUMAN	This API	A reviewer on your team said this.
SCORE_SOURCE_JUDGE / SCORE_SOURCE_EVAL	Automated evaluators	A model said this. Read-only.
SCORE_SOURCE_HARNESS	A dataset run	A harness measured this.

An average across sources is a misleading number

A 1-5 from an end user and a 1-5 from a domain reviewer are different measurements. Blending staff opinion into end-user sentiment corrupts the signal silently and irreversibly — and end-user sentiment is what feeds back into the product.

Always filter list-scores by source when aggregating. This is also why reviewer ratings are recorded here and never through the message-rating API: an end user's rating changes what the agent says to them next, and a review must have no side effect on the thing under review.

Comments

Comments attach a note to a trace, observation or session, so context travels with the artifact instead of in a side channel:

curl -X POST https://api.travila.ai/api/v1/evals/create-comment \
  -H "X-API-Key: sk_your_key_here" -H "X-On-Behalf-Of: reviewer@yourcompany.com" \
  -H "Content-Type: application/json" \
  -d '{"objectType": "TRACE", "objectId": "…", "content": "Tool call returned stale data."}'

A comment sent with a bare API key is stored without an author. Send an identity if you want to know who said it.

Closing the loop

A reviewer who finds a bad turn should not stop at scoring it — add it to a dataset so it stays fixed. You already have the turn address, which is all add-dataset-item needs.

Human labels also have a second life: they are the ground truth an automated judge is calibrated against. A queue completed by people is what tells you whether a model-based judge can be trusted to score the rest.