Evals
Every LLM turn your agents run is recorded as a trace — the memory lookups, tool calls and generations behind the reply, not just the text. The Eval APIs let you read those traces, judge them, and turn the bad ones into a regression suite.
POST /api/v1/evals/<method>
X-API-Key: sk_your_key_here
Content-Type: application/json
These are observability and evaluation APIs, not agent tools. Nothing here is on the path of a live conversation: reading, scoring and curating never changes what an end user sees.
The two loops
The 28 endpoints only make sense as two workflows. Almost every question about which call to make next is answered by deciding which loop you are in.
| Datasets and runs | Review | |
|---|---|---|
| Who | An automated harness | A human reviewer |
| Asks | Did this change regress anything? | Was this specific reply any good? |
| Shape | Curate a golden set, replay it, record the run | Define dimensions, queue turns, score them |
| Auth | Works with a bare API key | Needs a user identity — see below |
They meet in the middle: reviewers harvest bad turns into datasets, and dataset items need expected outputs that only a human can author.
Authentication, and the one place it bites
Send a tenant-scoped API key in X-API-Key. No user identity is required for most calls —
this API is built for machine callers that authenticate as a tenant rather than as a person.
record-score needs a user identity, a bare API key is not enoughA score has to be attributable to someone, so record-score reads the rater from the
verified caller and rejects a request that has none:
{"code": 401, "message": "no verified subject on the request context; a score must be attributable"}
Every other endpoint works with a bare sk_… key. Only scoring needs a person behind it —
either a console JWT, or an API key with
X-On-Behalf-Of.
This is deliberate: a rater id that a client could supply would let any caller write or clear someone else's rating.
create-comment behaves differently again — it accepts a bare key, but the comment is then
stored with no author. Send an identity if you want the comment attributed.
Projects
An optional X-Project-Id header selects which sub-tenant project you read and write. It
defaults to default, the only project until multi-project support ships. It scopes a
request; it does not authenticate one — your key still determines the tenant.
If the tenant and project you address has no evaluation backend provisioned, calls fail with 424 and a message naming both. That is a provisioning gap, not a bad request — retrying will not help, and it is the response to expect when pointing a harness at a tenant for the first time.
Traces are redacted
Trace content passes through PII redaction before it is stored, and the conversation store's content does not. The same turn reads differently in the two places, and the trace is the lossy one.
Read operations return contentRedacted so you can tell. When it is true, present the
text as redacted rather than as verbatim — otherwise a reviewer comparing a trace against
the conversation will report the difference as data loss.
This matters beyond display: dataset items harvested from traces capture redacted context, so goldens replay redacted inputs and judges score redacted ground truth.
Addressing a turn
You will rarely have a trace id. Callers hold messages, so the API takes the turn's address instead and derives the id server-side:
{"conversationId": "conv_123", "sourceUserMessageId": "msg_abc"}
sourceUserMessageId is the id of the user message that opened the turn, and every
message in that turn carries it:
| You have | sourceUserMessageId is |
|---|---|
| The user message that started the turn | its own messageId |
| Any assistant message from that turn | its sourceUserMessageId |
| A rating event | its sourceUserMessageId |
One turn is one trace, even when the agent went through several rounds of tool calls. All of those generations nest inside it.
Reading traces
list-traces filters by conversation, profile, end user, session, time range, tags, name
and level. get-trace takes either a traceId or the turn address above — exactly one.
Two fields on a trace are worth knowing about:
profileId— which agent profile produced the turn.configHash— a content hash of the resolved generation config (system prompt, model, parameters). This is what makes "did quality move when we changed the prompt?" answerable: filter scores by it and you are comparing prompt revisions, not just profiles.
Pagination
List operations take page (1-based) and pageSize, and return a page object with
totalItems and totalPages. There is no cursor. 64-bit integers arrive as JSON strings,
per the protobuf JSON mapping.
Where to go next
- Datasets and runs — build a golden set and record what a harness scored against it.
- Review — define score configs, queue turns for human review, and record ratings and comments.
- Eval API reference — every endpoint, field by field.
Endpoint summary
All 28 endpoints, grouped by what they are for. Every one is POST /api/v1/evals/<method>.
| Group | Endpoints |
|---|---|
| Traces and observations | list-traces, get-trace, list-observations |
| Sessions | list-sessions, get-session |
| Scores | list-scores, record-score, delete-score |
| Score configs | list-score-configs, create-score-config, update-score-config |
| Comments | create-comment, list-comments, delete-comment |
| Datasets | list-datasets, create-dataset, add-dataset-item, list-dataset-items, update-dataset-item |
| Dataset runs | list-dataset-runs, record-dataset-run, get-dataset-run |
| Annotation queues | list-annotation-queues, create-annotation-queue, enqueue-for-annotation, list-annotation-queue-items, complete-annotation-queue-item |
| Dashboard | get-overview |