Evals
Every scorer answers the same way: a score from 0 to 1, a one-word label, and a reason a person can read. Datasets, experiments and online scoring all use the same scorers, so a number means the same thing wherever it appears.
Datasets
A dataset is a list of items, each with an input, an optional expected answer and optional assertions. Add items from a real run (one click on the run page or in the review queue), from saved fixtures, or from a CSV. Runs only become items when their content was captured; see content.
Experiments
An experiment replays a dataset through up to four variants, each a prompt or a model, and scores every answer. The first variant is the baseline, and every other variant is reported as a delta against it. An experiment runs up to 200 items.
Read only. Replay strips every tool that writes, so an experiment on a production run cannot open a PR, send a message or change a record. The experiment page says which tools were stripped.
Scorers
| Scorer | What it checks |
|---|---|
claims | Checks the claims a report makes against the tool calls the run actually made. Catches an agent that says it opened a PR it never opened. The default for online scoring. |
assertions | The assertions stored on the dataset item, evaluated against the output and the tool calls. |
exact | The output equals the expected answer. |
contains | The output contains the expected answer. |
regex | The output matches a pattern; without one, the expected answer is used as the pattern. |
json | The output parses as JSON and carries the top-level keys you name. |
judge | A model grades the answer on your rubric; without one, on correctness and completeness. The judge’s own spend is recorded with the score. |
Online scoring
Set a sample rate on a project and that share of its live runs is scored as it lands, with the scorers you pick (the claims scorer by default). The rate is capped at 20 percent. A run whose project does not capture content is skipped rather than scored as a failure.
Review
Low-scoring and flagged runs collect in one queue. Each row takes a thumbs up or down, a label and a note, and can be added to a dataset, which is how a production failure becomes a regression test.
Which tier has what
Datasets (one on the free tier), manual experiments and every scorer are on every tier. Online scoring and the review queue need e-volv Observer on its paid tier or Studio; see tiers and retention.