Build benchmark mode back end

Epic 12243 (plan task 12246). Full settled inputs: the PRD (Task 12244) and the Architecture (Task 12245) of this epic — read them with `task_read`/`artifact_read` on those tasks; section numbers below (R…/AC…/§…) cite them. Where PRD and the original proposal (Task 12195) differ, the PRD governs. Builds on the ledger writer/tick, the outcome scan's CI read and the metrics `ratings()` stub. ## Original requirement (verbatim) > "6. Benchmark mode (last phase): plan feature + per-project opt-in, budget-capped, default sample 5% of coder tasks: clone a sampled task to a shadow task on an alternative configuration in its own worktree; the shadow PR is never merged (closed with a body naming the duel); gate pass, blind-review verdict and diff distance to the merged PR give a pairwise result; a Bradley-Terry rating per configuration is shown beside the observational metrics." (Settled: D10 budget = share of project's monthly agent spend, default 10%, knob `exploration_budget_share` shared with routing; D12 no plan carries it — operator org override only; dogfood Jaah org + Jaah project in dev.) > "7. Measurement is always collected (ledger, metrics, pages); only benchmark mode changes behaviour and only behind its plan feature and project opt-in. Dogfood on the Jaah project first." > Done means (epic, verbatim): "With benchmark mode ON for a project, a sampled task produces a shadow task, a closed unmerged shadow PR, one pairwise record and an updated rating; with it OFF or the plan term absent, no shadow task exists." PRD requirements carried (verbatim): R6.1 "There is a plan switch `benchmark_mode` in `SWITCHES` (`apps/api/app/plan_terms.py:186-199`). It is OFF on every plan and enabled only by an operator org override (D12). There is also a per-project opt-in toggle in project settings, visible only when the org's switch is ON." R6.2 "When both are ON, the CP samples coder tasks at the project's sample rate (default 5%). For each sampled task it creates a shadow task on an alternative allowed configuration, in its own worktree." R6.3 "The budget is shadow dollars per calendar month, capped at a share of the project's monthly agent spend (default 10%, one knob shared with the routing epic, D10). Once the cap is reached, no new shadow is created that month, and the project settings show \"Budget reached\"." R6.4 "The shadow PR is never merged. It is closed with a body naming the duel (origin task and both configurations). Shadow rows are flagged and excluded from the observational metrics." R6.5 "Each duel produces a pairwise record from three inputs: gate pass, blind-review verdict and diff distance to the merged PR. A Bradley-Terry rating per configuration is recomputed and shown beside the observational metrics on Performance." R6.6 "With the switch OFF, or the project opt-in OFF, no shadow task is ever created." R7.2 "Outside benchmark mode, dispatch, placement, agent resolution and merge behaviour are unchanged." ## Approach (settled — realize, do not reopen) Architecture §3.5, §12 (all), Pinned seams 2 (`benchmark.verdict`), 4, 5; architecture decisions A6 (least-dueled allowed challenger) and A7 (hidden `duel_review` runner). - One additive migration: `benchmark_duels` + `projects.benchmark_opt_in` (default false), `benchmark_sample_rate` (0.05), `exploration_budget_share` (0.10); org-transfer rule. `benchmark_mode` plan switch in `SWITCHES` (default False everywhere) + `CONSOLE_SWITCHES` + web twin `lib/planSwitches.tsx`. - Sampler in `outcome_ledger_tick` (e): deterministic `hash(task_id) mod 10000 < rate × 10000`; budget check vs this month's non-shadow agent spend; in-flight duels finish. - Challenger: allowed set (built-in providers with ≥ 1 enabled org account + runnable SDK agents) × coder tiers max/high/standard; minus the origin config; fewest decided duels vs origin config, random tie-break; set of size 1 samples nothing. - Shadow task: hidden `coder` child of a hidden `benchmark` parent (never in the origin's epic), copy of origin body, `shadow=true`, pins via existing `agent_id`/`bound_provider`/`model` (no resolution-order change); brief appendix → draft PR `[benchmark] …`, never enqueue/merge, settle new terminal `shadow_pr` (pr_url required, shadow tasks only, in `task_manifest.TERMINAL_TYPES`). - Merge guard: launch payload `shadow: true` → `JAAH_TASK_SHADOW=1`; `tools/check_gh_pr_guard.py` denies `pr merge`/`pr auto-merge`/`pr ready` when set; CP refuses `pull_request_enqueue` and `merged_pr` for shadow tasks; `merge_lane.report_pr` refuses a shadow PR. - `duel_review` hidden analysis persona + toolbox `tools/duel_review.py` (plain git diffs A/B in stored random order, no author/branch lines; normalised line-Levenshtein `diff_distance`); placed on a provider different from both duelists when possible; reports via `benchmark.verdict`; closes the shadow PR with `jaah-forge pr close` and a body naming origin task, both configs, duel id; `duel_close` retry on failure. - Pairwise result: gate decisive when exactly one passed → else review verdict → else tie (distance < 0.3 keeps tie). BT: MM fit per org, ½–½ ties, one virtual tie prior, 200-resample bootstrap 95% interval, ≤ 500 duels, cached 10 min; replaces the `ratings()` stub so `profiles.bt` is filled. Shadow rows excluded from every observational metric; duel cost counts toward the budget only. Credentials & reach: - pytest ACs: none beyond the worktree (`./dc` stack) — verified (worktree). - Live dev DB reads: `tools/cp_psql.sh dev` → broker refused `org/aws/cp-psql-reader` — not needed by any oracle (every AC is a UI flow or a test). Out of scope: the project-settings benchmark controls, BT column, duel link and the dev dogfood toggles (benchmark UI child — the toggles need its UI); any change to the origin task's dispatch; `agent_resolution.py` order; routing. Out of scope (epic, verbatim): "any routing behaviour (the routing epic reads this ledger); new production signal sources (the production QA epic writes deployed/incident_linked); a public index or tenant-to-tenant sharing; re-scoring tasks older than a one-off backfill of merge, lane and cost facts." Do NOT touch (PRD A8): no change to `apps/api/app/agent_resolution.py` resolution order, `apps/api/app/jaah_scheduler.py` account placement, or `apps/api/app/jaah_routing/` (`routing_decision_id` stays null); no deploy ingestion, monitor adapters, incidents table or new trigger source kinds (`apps/api/app/triggers.py` untouched; `deployed_at`/`incident_linked` are columns/states nothing writes); no public page, no unauthenticated endpoint, no tenant-to-tenant view, no tenant name on any operator-plane aggregate; no retired or rewritten weekly outcome-review job (`apps/api/app/operator_dashboard/review.py`, `apps/api/app/operator_dashboard/quality.py`, D4); backfill writes merge, lane and cost facts only, never re-runs LLM scoring, never re-opens old tasks; benchmark mode never merges a shadow PR and never changes the origin task's dispatch. No CP forge call (ADR-0156): forge facts come only through the `outcome_scan` runner. Migrations additive only (new tables + nullable/defaulted columns). Done means: - AC6.2 "A test with the switch OFF, the opt-in OFF, or the budget exhausted runs the sampler over 100 coder tasks and asserts zero shadow tasks. Run `pytest apps/api -k benchmark` and observe it pass." Also: with the plan term absent from the plan (no override) zero shadows. - AC6.1 back-end half (test-level): with both ON, a sampled origin produces exactly one hidden shadow task pinned to the challenger, its `shadow=true` row excluded from metrics; a `shadow_pr` settle is accepted only for shadow tasks; `pull_request_enqueue`, `merged_pr` and `merge_lane.report_pr` are refused for it; the guard denies `pr merge`/`pr auto-merge`/`pr ready` under `JAAH_TASK_SHADOW=1`; a `benchmark.verdict` report produces one decided duel with the gate→review→distance rule applied; `profiles` returns updated `bt` for both configurations. - BT fit unit test against hand-computed ratings on a small fixture; ties counted ½–½; finite at n=1. - R7.2: existing dispatch, placement, resolution and merge tests pass unchanged (`./dc check full`). - Gate (epic Done means, verbatim: "Required checks: ./dc check quick green; changed-area tests green; merge lane full gate green.") — `./dc check quick` green; changed-area pytest/vitest green; merge lane `./dc check full` green; Jaah Deploy of the PR succeeds.