Review and score turns

Section ID: Guide/jctgnv5fj4#review-and-score-turns

Review replies that need a person’s judgment, such as an answer that is accurate but hard to use. Give reviewers a common rating scale, queue the turns and record both scores and explanations. Use the reviewed examples to improve regression datasets and calibrate automated judges.

Warning: This loop needs a user identity

record-score rejects a bare API key because the score must be attributable to a reviewer. Use an authorized public API key with verified X-On-Behalf-Of; console JWTs belong to the separate console surface. Score configs, queues and comments accept a bare key. See record-score for the exact contract.

The loop

Section ID: Guide/jctgnv5fj4#the-loop

Build a weekly review queue for coaching replies whose helpfulness needs a person’s judgment. You need evaluation enabled, a backend credential, a verified reviewer identity for scoring, and trace IDs from the investigation recipe.

1. Define a shared helpfulness scale and an explanation of what each rating means.

2. Create a queue using that score configuration and add the selected traces.

3. Let the reviewer read a queued trace, record the score and explain it.

4. Complete the queue item only after the review is saved.

5. Turn a useful failure into a curated regression case.

The result is a completed work item plus an attributable judgment. Completing the queue is a separate operation from saving the score; neither guarantees that the judgment is correct.

Record the score and any explanatory comment before completing the queue item. These are separate actions:

1. Create the score configuration
2. Create the annotation queue
3. Enqueue items
4. List queue items
5. Record a score
6. Create a comment
7. Complete the annotation queue item

Keep reviewer judgments separate from user sentiment

Section ID: Guide/jctgnv5fj4#never-blend-score-sources

Keep customer sentiment, expert review and automated measurements separate. Choose the score source for the question you are answering before computing an aggregate; use reviewer scoring for this workflow.

1. Define the dimensions first

Section ID: Guide/jctgnv5fj4#1-define-the-dimensions-first

A score config is a rating scale. Reviewers pick from it rather than inventing their own, which is what makes two reviewers' labels comparable.

Request example: Create a score config. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Create and save a numeric or boolean trace-scoring configuration before recording ratings. Use the supported write and configuration contract when designing a different scale.

2. Queue what needs reviewing

Section ID: Guide/jctgnv5fj4#2-queue-what-needs-reviewing

Save the returned configuration ID and replace cfg_helpfulness below with it. A queue is a worklist bound to a fixed set of configs, so everyone reviewing from it scores the same dimensions. create-annotation-queue requires at least one entry in scoreConfigIds — a queue with no configs is rejected.

Request example: Create an annotation queue. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Save the returned queue ID and replace q_weekly in later requests. Enqueue the trace you want reviewed, using EVAL_TARGET_TYPE_TRACE for the currently qualified score path. The target-type contract lists other target kinds; their presence in the enum does not qualify them for this scoring recipe.

Request example: Enqueue an item for annotation. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Pull the worklist

Section ID: Guide/jctgnv5fj4#pull-the-worklist

Request example: List annotation queue items. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Response:

Response example: List annotation queue items. Example identifiers and values are illustrative; substitute authorized values.

Open the queued trace and retain separate evidence when you need durable completion attribution; see queue addressing and completion attribution.

3. Record the rating

Section ID: Guide/jctgnv5fj4#3-record-the-rating

Read the trace and apply the configured rubric before submitting the score. Use the reviewer’s verified identity and the saved score-config ID. After saving, read the stored score; serialize repeat actions so a delayed request cannot silently become a second review.

Request example: Record a score. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; X-On-Behalf-Of: reviewer@yourcompany.com; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Reconcile the stored score after repeated actions; see reviewer identity and replacement limits.

Save the explanation with the reviewed turn

Section ID: Guide/jctgnv5fj4#comments

Explain why the reply received its rating, such as a tool returning stale data. Attach the comment to the same reviewed trace and supply the reviewer’s identity so another person can understand who made the observation:

Request example: Create a comment. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; X-On-Behalf-Of: reviewer@yourcompany.com; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Keep the returned comment ID, then read the trace’s comments to confirm the explanation was saved before completing the review. Keep the note relevant and under the 3000-character limit; longer content is rejected. A bare-key comment has no author.

The comment reference defines other supported targets. For this queue, continue using the trace you scored. If an authorized administrator needs to remove a note, delete-comment suppresses it from the list while retaining the record for audit. It does not erase every retained copy.

4. Complete the saved review

Section ID: Guide/jctgnv5fj4#mark-it-done

Call this after recording the rating and any explanatory comment, not when the reviewer merely opens the task. If recording failed, leave the item pending while you recover the score. Keep separate evidence if you need durable completion-author attribution: the current completion operation does not retain it.

Request example: Complete an annotation queue item. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; X-On-Behalf-Of: reviewer_9; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Response:

Response example: Complete an annotation queue item. Example identifiers and values are illustrative; substitute authorized values.

Completing is what stops an item being handed out again — it is not implied by recording a score. A reviewer who scores but never completes leaves the item in the pending pool for the next person.

Closing the loop

Section ID: Guide/jctgnv5fj4#closing-the-loop

A reviewer who finds a bad turn should not stop at scoring it — add it to a dataset so it becomes an explicit regression case. Supply the input and reviewed expectation as described in dataset curation; the turn address alone does not capture the context.

Human review can provide calibration examples for an automated judge. Use a clear rubric, resolve reviewer disagreements and evaluate the judge on separate examples. Completing a queue alone does not establish that its labels are correct or that the judge will be reliable on other cases.

Choose a different rating scale

Section ID: Guide/jctgnv5fj4#score-config-types-in-depth

The recipe uses a 1–5 helpfulness rating. Use boolean trace scoring when a simple pass/fail judgment better matches the criterion. The schema also names categorical/text and other target types, but the current score write path supports numeric/boolean values on traces only.

Numeric bounds are UI hints, not enforced score limits; validate the reviewer’s choice in your application. The data type and categorical values cannot change after creation. For a different scale, create a new configuration so earlier scores retain their meaning. See the score-configuration reference for exact settings.

Retire a rubric without reinterpreting earlier scores

Section ID: Guide/jctgnv5fj4#list-and-retire-configs

When the helpfulness rubric changes, list its configurations, archive the old one through update-score-config, and create a replacement. Configurations are archived rather than deleted so earlier scores keep their reference.

Include archived configurations when displaying historical reviews. Changing a used configuration’s numeric bounds does not rescale its earlier scores; prefer a replacement when the meaning of the scale changes. Use the returned new configuration ID for future queues and scores.

Return to an existing review queue

Section ID: Guide/jctgnv5fj4#find-the-queues

Save the queue ID created for weekly review. When reviewers return, use that identity to load its pending items rather than creating another queue.

If the application needs a queue picker, use list-annotation-queues and its pagination contract. Confirm the chosen queue’s rubric before presenting its work; a similarly named queue can use a different scoring configuration.

Withdraw a rating

Section ID: Guide/jctgnv5fj4#withdraw-a-rating

Request example: Delete a score. Example identifiers and values are illustrative; substitute authorized values.

Example request headers: X-API-Key: sk_your_key_here; X-On-Behalf-Of: reviewer@yourcompany.com; Content-Type: application/json. Replace the credential and user placeholders with your own authorized values.

Response:

Illustrative response shape; omitted default fields and actual outcomes depend on the operation.

Response example: Delete a score. Example identifiers and values are illustrative; substitute authorized values.

Use trusted administrative tooling for this operation and read back the result, allowing for delayed views. The score deletion contract explains the current authorization limit.

Document ID: Guide/jctgnv5fj4