Benchmark ratings show dishonest intervals at small
A configuration rated from a single benchmark duel shows a zero-width 95% interval on Command center › Performance › Configuration, for example `869 [869–869]` and `1131 [1131–1131]`. That reads as perfect certainty when it is the least certain a rating can be. Seen while building the Duel rating column (Task 12288).
Likely cause: `benchmark.compute` (apps/api/app/outcome_ledger/benchmark.py) bootstraps by resampling the duel list. With one duel, every resample is that same duel, so every draw is identical.
Ask:
- Make the interval honest at small n. Either derive it so one duel cannot give a zero-width interval (for example, keep the virtual-tie prior in each resample, or use a likelihood-based interval), or return `lo`/`hi` as null below a minimum number of duels so the console shows the rating without an interval.
- Keep the wire shape `{rating, lo, hi, n_duels}`. The console already renders `—` when the whole rating is null.
Out of scope: the sampler, duel filing, guards, and the console column itself (built in Task 12288).
Done means:
- A unit test in apps/api/tests/test_benchmark_mode.py shows that a configuration rated from one duel has `lo < rating < hi`, or `lo`/`hi` null per the chosen rule.
- The existing hand-computed BT tests still pass.
- Required checks: ./dc check quick green; changed-area tests green.
Outcome
Already shipped by Task 12442: develop returns null lo/hi for a one-duel rating, and test_benchmark_mode passes (37). I dropped my Wald-interval version because it conflicted with that fix. No PR. Memory: none applied.
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)