Measure and improve quality
Section: DOC-IS-evaluation#measure-and-improve-quality.
Improve an answer your customers struggle to use, and keep that improvement from breaking other answers. Start with one real reply, have a reviewer explain the problem, then compare a change against the same known examples.
Choose the problem to solve
Section: DOC-IS-evaluation#start-here.
| The problem in your application | Recipe |
|---|---|
| A reply was slow, failed or received a poor rating | Investigate the turn and its recorded steps |
| Your team needs to judge replies consistently | Run a review queue with a shared rubric |
| A prompt or model change might break a previously good answer | Compare both versions on a regression dataset |
| Users need to say whether a reply helped | Add message feedback to the conversation |
Before you start
Section: DOC-IS-evaluation#projects.
Have evaluation enabled for your account, permission to inspect the selected conversations, and controlled test data and spending for any reruns. An HTTP 424 means evaluation is not provisioned; ask your operator to enable it rather than repeating the request. The current API uses the legacy default project, without separate project, test/live or customer isolation created by a header.
Use the account’s supported API version. The trace recipe explains redaction and collecting complete results; the review recipe explains reviewer identity and current mutation limits. Use the Evaluation API reference for exact operation fields.
Turn one poor reply into a tested improvement
Section: DOC-IS-evaluation#the-two-loops.
- Find the reply’s recorded steps and identify what went wrong. Keep missing or redacted context visible; absent telemetry does not prove an action never occurred.
- Review the answer against a shared rubric and save the reason for the judgment. Keep customer sentiment, expert review and automated scores separate.
- Create a regression case with the input needed to reproduce the problem and an expert-reviewed expectation or objective validator. An observed model answer is not automatically ground truth.
- Run the baseline and candidate against the same eligible cases in your test harness. Constrain tools and spending; recording a dataset run does not execute its cases.
- Compare actual outcomes with failed, excluded and missing cases visible, then decide whether to adopt the change. Save the configuration and results with that decision.
You finish with a specific problem, its reviewed explanation and a comparison another reviewer can inspect. The current API stores evaluation evidence; it does not automatically change the serving prompt or train a model. The recipes cover the current workflow; the additions below describe the upcoming review and comparison experience.
Upcoming: revise a review without losing its history
Status: Upcoming — not yet available.
Section: DOC-IS-evaluation#review-provenance
Open the reviewed item and rubric version, edit under your own reviewer identity, and save. If the review changed meanwhile, resolve the conflict against the current version. The history keeps the author, rubric and earlier judgments attached to the item, with customer opinion, expert annotation and automated results distinguishable. Withdrawal requires the permitted author or administrator action.
These revision and ownership controls are not available in the current scoring and comment APIs. Keep current score deletion in trusted administrative tooling as described in review withdrawal.
Upcoming: repeat a release comparison from saved versions
Status: Upcoming — not yet available.
Section: DOC-IS-evaluation#reproducible-comparisons
Select a versioned dataset, saved baseline and candidate configurations, and scoring rubric. Run both on the same eligible cases, then inspect per-item scores and the evaluated, failed, excluded and missing counts. Review incomplete coverage and uncertainty before approving the change. Keep those versions with the decision so another reviewer can repeat the comparison.
All permitted result pages remain available to inspect, and a deleted result keeps an explicit outcome. For the current API, save the full harness configuration and reconcile outcomes using the dataset recipe; mutable dataset records or a supplied profile hash alone do not capture a reproducible experiment.
Document ID: DOC-IS-evaluation. Section identities and revisions.