Benchmark ratings show dishonest intervals at small

Task 12439 · Done · Coder · created · updated

A configuration rated from a single benchmark duel shows a zero-width 95% interval on Command center › Performance › Configuration, for example `869 [869–869]` and `1131 [1131–1131]`. That reads as perfect certainty when it is the least certain a rating can be. Seen while building the Duel rating column (Task 12288). Likely cause: `benchmark.compute` (apps/api/app/outcome_ledger/benchmark.py) bootstraps by resampling the duel list. With one duel, every resample is that same duel, so every draw is identical. Ask: - Make the interval honest at small n. Either derive it so one duel cannot give a zero-width interval (for example, keep the virtual-tie prior in each resample, or use a likelihood-based interval), or return `lo`/`hi` as null below a minimum number of duels so the console shows the rating without an interval. - Keep the wire shape `{rating, lo, hi, n_duels}`. The console already renders `—` when the whole rating is null. Out of scope: the sampler, duel filing, guards, and the console column itself (built in Task 12288). Done means: - A unit test in apps/api/tests/test_benchmark_mode.py shows that a configuration rated from one duel has `lo < rating < hi`, or `lo`/`hi` null per the chosen rule. - The existing hand-computed BT tests still pass. - Required checks: ./dc check quick green; changed-area tests green.

Outcome

Already shipped by Task 12442: develop returns null lo/hi for a one-duel rating, and test_benchmark_mode passes (37). I dropped my Wald-interval version because it conflicted with that fix. No PR. Memory: none applied.

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)