Add benchmark mode console controls
Epic 12243 (plan task 12246). Full settled inputs: the PRD (Task 12244) and the Architecture (Task 12245) of this epic — read them with `task_read`/`artifact_read` on those tasks; section numbers below (R…/AC…/§…) cite them. Where PRD and the original proposal (Task 12195) differ, the PRD governs. Builds on the benchmark back end, the Command center Performance page and the task Outcome panel.
## Original requirement (verbatim)
> "6. Benchmark mode (last phase): plan feature + per-project opt-in, budget-capped, default sample 5% of coder tasks: clone a sampled task to a shadow task on an alternative configuration in its own worktree; the shadow PR is never merged (closed with a body naming the duel); gate pass, blind-review verdict and diff distance to the merged PR give a pairwise result; a Bradley-Terry rating per configuration is shown beside the observational metrics."
(Settled: D10 budget = share of project's monthly agent spend, default 10%, knob `exploration_budget_share` shared with routing; D12 no plan carries it — operator org override only; dogfood Jaah org + Jaah project in dev.)
> Done means (epic, verbatim): "With benchmark mode ON for a project, a sampled task produces a shadow task, a closed unmerged shadow PR, one pairwise record and an updated rating; with it OFF or the plan term absent, no shadow task exists."
PRD requirements carried (verbatim): R6.1 "There is also a per-project opt-in toggle in project settings, visible only when the org's switch is ON." R6.3 "...the project settings show \"Budget reached\"." R6.5 "A Bradley-Terry rating per configuration is recomputed and shown beside the observational metrics on Performance." R6.7 "Dogfood: in dev at release, the Jaah org's override is ON and the Jaah project's opt-in is ON."
## Approach (settled — realize, do not reopen)
Architecture §9.3 settings UI and §12 dogfood.
- Project settings: benchmark opt-in, sample rate, budget share, month-to-date shadow spend vs cap and the "Budget reached" state — rendered only while the org's `benchmark_mode` is ON.
- Org overrides form: `benchmark_mode` override visible to the operator (if the generic switch override UI doesn't already render it).
- Command center › Performance: BT rating column (rating, interval, n duels) beside the observational metrics; grey when no duels.
- Origin task Outcome panel: duel link (shadow task, shadow PR, result) on a sampled origin; Shadow badge on the shadow task.
- Dogfood: through the console UI on dev only (CLAUDE.md hard rule — no SQL, no API script): set the Jaah org's `benchmark_mode` override ON and the Jaah project's opt-in ON.
Credentials & reach:
- pytest ACs: none beyond the worktree (`./dc` stack) — verified (worktree).
- Dev UI flows as Jaah-org Owner and as operator: `me/jaah/debug/main/operator-login` (operator, switches into tenants); environment `https://app.dev.jaah.ai` (public dev console; the internal `dev.jaah.ai` ALB did not answer from h2 within 15 s) — verified: credential_get ok; `app.dev.jaah.ai` → HTTP 200 from h2.
Out of scope: the sampler, duel, guard and BT logic (back-end child); turning benchmark on for any org other than Jaah.
Out of scope (epic, verbatim): "any routing behaviour (the routing epic reads this ledger); new production signal sources (the production QA epic writes deployed/incident_linked); a public index or tenant-to-tenant sharing; re-scoring tasks older than a one-off backfill of merge, lane and cost facts."
Do NOT touch (PRD A8): no change to `apps/api/app/agent_resolution.py` resolution order, `apps/api/app/jaah_scheduler.py` account placement, or `apps/api/app/jaah_routing/` (`routing_decision_id` stays null); no deploy ingestion, monitor adapters, incidents table or new trigger source kinds (`apps/api/app/triggers.py` untouched; `deployed_at`/`incident_linked` are columns/states nothing writes); no public page, no unauthenticated endpoint, no tenant-to-tenant view, no tenant name on any operator-plane aggregate; no retired or rewritten weekly outcome-review job (`apps/api/app/operator_dashboard/review.py`, `apps/api/app/operator_dashboard/quality.py`, D4); backfill writes merge, lane and cost facts only, never re-runs LLM scoring, never re-opens old tasks; benchmark mode never merges a shadow PR and never changes the origin task's dispatch. No CP forge call (ADR-0156): forge facts come only through the `outcome_scan` runner. Migrations additive only (new tables + nullable/defaulted columns).
Done means:
- AC6.1 "Dev flow. With the Jaah project opted in, wait for or force-sample a coder task. Observe a shadow task linked from the origin, and a shadow PR in state Closed (not merged) whose body names the duel. Agents › Performance shows one more pairwise record and an updated rating for both configurations."
- Forcing the sample (no API/SQL nudge): in the Jaah project's settings set the sample rate to 100%, let the next coder task reach merged, observe its duel decide, then restore 5% via the UI; record both changes and times in the report.
- AC6.3 "As an org Owner of a non-overridden org, project settings show no benchmark toggle."
- R6.7: on `https://app.dev.jaah.ai` the Jaah org's override and the Jaah project's opt-in read ON, set via the UI.
- At 390 px the new settings and the BT column have no horizontal page scroll; vitest covers toggle visibility by switch state and the "Budget reached" state.
- Gate (epic Done means, verbatim: "Required checks: ./dc check quick green; changed-area tests green; merge lane full gate green.") — `./dc check quick` green; changed-area pytest/vitest green; merge lane `./dc check full` green; Jaah Deploy of the PR succeeds.
Activity
- Draft
- Ready