# Evaluation release brief

This fictional release brief compares two navigation models on the same held-out scenes. The numbers are illustrative, not measurements of a deployed model.

## Decision and evidence

1. **Freeze the evaluation set** before comparing candidates.

   Keep the scene manifest, seed and episode count identical. Exclude training scenes and record every exclusion so another reviewer can reproduce the comparison.

   1. Check scene overlap against the training manifest.
   2. Save the final manifest with the evaluation report.
2. **Compare success and latency together.** A higher success rate does not justify missing the device's response budget.
3. **Publish the decision** with failure examples and a rollback owner.

## Compact checks

- [x] Same simulator version
- [x] Same device and batch size
- [ ] Failure clips reviewed

## Numbered release checklist

1. [x] Freeze the candidate checkpoint.

   Record the model hash and configuration beside the manifest. A filename alone is not enough to reproduce the run.
2. [ ] Review the comparison with the release owner.

   Inspect collisions, timeouts and slow responses separately. Preserve the baseline even when the candidate wins on the aggregate metric.
3. [ ] Approve a staged rollout.

   Set the rollback trigger before enabling the candidate. The last paragraph should end at the same content inset as the first.

## Comparison grid

::::grid{columns=2}
:::col
### Baseline

92% success across 500 held-out episodes; 84 ms median response time on the test device.

- Stable on open corridors
- Misses some narrow doorways
:::
:::col
### Candidate

95% success on the same episodes; 89 ms median response time.

- Better doorway recovery
- Needs review for slow turns
:::
:::col
### Go criteria

1. [x] No scene overlap
2. [ ] No new safety failures
3. [ ] Response budget met
:::
:::col
### Rollback plan

Owner: release engineer. Restore the baseline if collisions increase or the response budget is exceeded in two consecutive evaluation windows.
:::
::::

The grid should read left to right on a wide memo and top to bottom on a narrow one. Exporting the memo keeps the Markdown source intact.
