Add outcome metrics engine and APIs
Epic 12243 (plan task 12246). Full settled inputs: the PRD (Task 12244) and the Architecture (Task 12245) of this epic — read them with `task_read`/`artifact_read` on those tasks; section numbers below (R…/AC…/§…) cite them. Where PRD and the original proposal (Task 12195) differ, the PRD governs. Depends on the ledger core child (tables incl. `outcome_links`).
## Original requirement (verbatim)
> "4. Metrics per configuration (agent × model family × persona × review tier) with roll-ups per agent, model, project, org: success rate (survived ÷ settled+failed, operator-cancelled excluded); first-pass acceptance (merged with zero lane failures, zero re-dispatch, zero human fix commits ÷ merged); defects introduced per 100 merges; rework (lane failures + re-dispatches + post-enqueue pushes + human fix commits per merged task, plus rework hours); completion time ready→merged median/p90 and agentic time; cost per successful task (Σ cost of all attempts ÷ survived); security issues per 100 tasks, credited to the detecting lane and debited to the authoring configuration. Wilson 95% intervals; a configuration ranks only at ≥ 20 outcomes, else \"insufficient data\"; stratify by persona and a complexity bucket (diff size, files touched, repo language, clarity score)."
> "5. Surfaces (console UI style guide): Agents › Performance profile per agent/model/configuration; Compare 2–4 configurations on identical strata; org Leaderboard (filters persona, project, period, min n); task Outcome panel shows the ledger row and attribution links; Command center widgets read the ledger in place of the weekly sample; CSV export per org. Operator plane: Fleet › Agent index with cross-org aggregates per configuration, each cell ≥ 5 orgs and ≥ 100 outcomes, never a tenant name."
(Settled: D4 show both — ledger widgets beside the unchanged `quality.avg`; D8 Owner/Admin only; Architecture A2 — the pages are tabs of the Command center, not under the grid-gated Agents section; the operator page is "Agent index" in the operator sidebar group.)
> Done means (epic, verbatim): "Fleet › Agent index refuses a cell under 5 orgs or 100 outcomes."
PRD requirements carried (verbatim): R4.1 "Seven metrics with exactly the formulas quoted above. The denominator for success rate is settled + failed + abandoned rows. Operator-cancelled rows are excluded." R4.2 "A configuration is agent × model family × persona × review tier. Roll-ups are per agent, per model, per project and per org." R4.3 "Every rate carries a Wilson 95% interval. A configuration with fewer than 20 outcomes is not ranked and shows \"Insufficient data\" in grey, with its n." R4.4 "Strata are persona × complexity bucket." R5.6 "CSV export per org: Owner/Admin only. One row per ledger row, including attribution-link labels. It never includes another org's data." R5.7 "A cell with fewer than 5 orgs or fewer than 100 outcomes shows \"Below threshold\" and no number. No org or tenant name appears anywhere on the page, in its API response or in its export." D8 Owner/Admin only.
## Approach (settled — realize, do not reopen)
Architecture §6.2, §9.1, §9.2, Pinned seams 3, 4, 5.
- `outcome_ledger/metrics.py`: one `rows(org_id | None, filters)` query; formulas exactly as §9.1 (final-rows denominator, first_pass, defects_per_100, rework + hours, completion median/p90 + agentic, cost_per_success null at 0 successes, security_per_100 wired to `task_outcomes.security_findings` — 0/null until the security child populates it); Wilson z = 1.95996 (per-100 rates on the ≥1-event proportion, scaled); `ranked = n_final ≥ 20`; every built-in provider (CLAUDE, CODEX, DEEPSEEK, ANTIGRAVITY, PI from `agent_providers.py`) always emitted for `group=agent`, even at n=0; strata `(persona, complexity_bucket)`; Compare on the intersection of strata where every config has ≥ 1 final row; empty intersection returns no metric values; shadow and excluded rows never counted; backfilled null facts drop the row from that metric only.
- Cross-org (`org_id=None`): a cell emits numbers only if distinct orgs ≥ 5 and n_final ≥ 100, else `{below_threshold: true}` with no n/counts/interval; totals summed over emitted cells only and labelled so; response model has no org field.
- Router `routes/outcomes.py` with the §9.2 routes: `profiles`, `compare`, `leaderboard`, `export.csv` (streamed, link labels `kind:state:confidence` joined by `;`), `GET /api/tasks/{id}/outcomes` (task visibility, `can_decide` flag), `GET /api/operator/agent-index` (`require_operator`). Tenant routes `require_org_admin`. Roll-ups (R4.2): `group=agent|model|config|project`, and the org roll-up is the unfiltered org scope. Config `label` is the full human name (e.g. "Claude Code · Opus · coder · Major"), never truncated.
- `outcome_ledger.benchmark.ratings(session, org_id) -> {}` stub so `bt` is null until the benchmark child.
- TS twin `apps/web/src/pages/command-center/outcomes/types.ts` pinned by a JSON fixture read by both pytest and vitest.
Credentials & reach:
- pytest ACs: none beyond the worktree (`./dc` stack) — verified (worktree).
- Live dev DB reads: `tools/cp_psql.sh dev` → broker refused `org/aws/cp-psql-reader` — not needed by any oracle (every AC is a UI flow or a test).
Out of scope: every page and widget (Command center tabs, Outcome panel, Agent index page are their own children); the BT fit (benchmark child); security findings capture.
Out of scope (epic, verbatim): "any routing behaviour (the routing epic reads this ledger); new production signal sources (the production QA epic writes deployed/incident_linked); a public index or tenant-to-tenant sharing; re-scoring tasks older than a one-off backfill of merge, lane and cost facts."
Do NOT touch (PRD A8): no change to `apps/api/app/agent_resolution.py` resolution order, `apps/api/app/jaah_scheduler.py` account placement, or `apps/api/app/jaah_routing/` (`routing_decision_id` stays null); no deploy ingestion, monitor adapters, incidents table or new trigger source kinds (`apps/api/app/triggers.py` untouched; `deployed_at`/`incident_linked` are columns/states nothing writes); no public page, no unauthenticated endpoint, no tenant-to-tenant view, no tenant name on any operator-plane aggregate; no retired or rewritten weekly outcome-review job (`apps/api/app/operator_dashboard/review.py`, `apps/api/app/operator_dashboard/quality.py`, D4); backfill writes merge, lane and cost facts only, never re-runs LLM scoring, never re-opens old tasks; benchmark mode never merges a shadow PR and never changes the origin task's dispatch. No CP forge call (ADR-0156): forge facts come only through the `outcome_scan` runner. Migrations additive only (new tables + nullable/defaulted columns).
Done means:
- AC4.1 "**Verify:** a fixture of known rows gives hand-computed values for all seven metrics and their Wilson bounds (to 3 dp). Run `pytest apps/api -k outcome_metrics` and observe it pass." (metric #7 hand-computed over fixture `security_findings` counts.)
- AC4.2 "**Verify:** the fixture has a configuration with 19 outcomes and one with 20. The first has `ranked = false`; the second has `ranked = true`."
- AC5.8 test half: "A test asserts the API omits a below-threshold cell's numbers (`pytest apps/api -k agent_index`)." — plus: the serialized JSON contains none of the seeded org names or ids, and no total lets a suppressed cell be derived by subtraction.
- AC5.2 API half: "Their APIs return 403." — a Member seat gets 403 from profiles, compare, leaderboard and export.csv; a non-operator gets 403 from agent-index.
- Compare: a test with two configs sharing strata returns `shared_strata` and metrics over the intersection only; two with none return `shared_strata: []` and no metric values.
- CSV: a test with two orgs asserts the export holds exactly this org's ledger rows, one per row, with link labels.
- `group=agent` on an empty org returns five provider groups with `insufficient: true` and full labels.
- The types fixture is read and passes in both `pytest` and `vitest`.
- Gate (epic Done means, verbatim: "Required checks: ./dc check quick green; changed-area tests green; merge lane full gate green.") — `./dc check quick` green; changed-area pytest/vitest green; merge lane `./dc check full` green; Jaah Deploy of the PR succeeds.
Activity
- Draft
- Ready