Paired evaluation#
Read a canonical evaluation report — two arms compared, with the isolation and evidence boundaries visible.
Before you start#
- The service running. Open
http://localhost:4287/evaluations. - A canonical evaluation report in the report slot. The viewer consumes one strictly validated report; it does not discover reports.
What the viewer does#
It displays a paired comparison: both arms, both seeds, and the direction of the delta, all explicit. Aggregate metrics and matched outcomes are shown against the evidence that produced them.
What it does not do#
The viewer never evaluates in the browser. It renders a report that was produced elsewhere. Nothing you do in this surface computes a result, and no model is called.
That is the property that makes the surface trustworthy: what you read is what the report contains, not a recomputation that might differ.
Strict validation#
The report is strictly validated before it is displayed. A report that does not conform is rejected rather than partially rendered, so you never read a half- valid comparison.
Reading a comparison honestly#
| Element | What to check |
|---|---|
| Arms | Both are named. A comparison with an implicit baseline is not a comparison |
| Seeds | Both are shown. Different seeds explain differences that are not treatment effects |
| Delta direction | Stated explicitly, so an improvement cannot be read backwards |
| Isolation boundary | What was held constant between arms |
| Evidence | The recorded outcomes behind each aggregate |
What a mock-only comparison proves#
A canonical mock-only comparison proves deterministic orchestration and export safety. It does not prove memory effectiveness or real-provider behaviour. Treat it as evidence that the machinery is sound, not as a result about model behaviour.
Verify#
- The viewer displays both arms, both seeds, and an explicit delta direction.
- Aggregate metrics are accompanied by the outcomes they aggregate.
If it does not work#
| Symptom | Check | Recovery |
|---|---|---|
| The viewer shows nothing | Whether a report occupies the canonical slot | Place a report in the slot; the viewer does not search |
| A report is rejected | The validation message | Strict validation refuses a non-conforming report rather than rendering part of it |
| Aggregates look implausible | Seeds and arms | Different seeds explain much; confirm before treating a delta as an effect |