Wire router to the outcome ledger

Task 12301 · Queued · Coder · created · updated

Epic 12251, child C9 (the PRD §6 ledger wiring slice, last). Needs C3b, C4, C5, C6a, C8 and — across epics — epic 12243's profile-phase coder task (ledger table `task_outcomes` and per-configuration profile reads merged on develop). That cross-epic edge could not be expressed at planning time (12243 had no children). PRECONDITION — check first: on develop, confirm 12243's ledger table and profile reads exist (e.g. grep apps/api/app for `task_outcomes` and the profile reader). If they have NOT merged, do not build ledger tables or metrics yourself: ask ONE round through AskUserQuestion with header `External`, naming the missing 12243 profile phase, and follow the charter's "When you cannot proceed". ## Original requirement (verbatim) Preamble: "Build the router over the outcome ledger. Depends on the outcome ledger epic (Task 12195 epic 1) reaching its profile phase." Ask 1 (part): "The ledger row links to it so every decision is graded by its outcome." Ask 2 (part): "per-configuration ledger statistics for the stratum, org first, operator-plane prior where the org has too little data." Ask 5 (part): "Their findings write the ledger's security-findings facts." Ask 6 (part): "rendered with its ledger profile … a listing's numbers are only the ledger's." Epic Done means (verbatim): "the scoreboard shows router vs default survival and cost over a period"; "In Auto on a project with two allowed providers, both receive dispatches over time with the exploration share bounded by the project knob"; "its findings land on the ledger row"; "The marketplace page shows each catalog agent with ledger numbers only". ## Settled scope (PRD §6, Architect C9) - (a) a real `LedgerStats` implementation in `agent_router/ledger_stats.py`: org first, operator-plane prior (aggregate only, no org name) when the org is below the minimum; import 12243's `MIN_OUTCOMES` constant so the threshold has one home; align `features.py` bucket thresholds with 12243's complexity bucket; `stats_version` reflects the ledger. - (b) the ledger row references `routing_decisions` by id: write `task_outcomes.routing_decision_id` (joined on `(task_id, attempt)` with `status='final'`); add the column only if 12243 left no room. - (c) blind review findings → the ledger's security-findings facts (skip if 12243's phase already reads `review_findings`; say so). - (d) scoreboard survival and cost-per-success with intervals and marketplace ledger numbers read the ledger; exploration budget sums switch to actual ledger cost where it exists; blind-child other-vendor choice becomes sampled among other-vendor candidates when data exists. Credentials & reach: - AC3/AC7 DB cross-checks: `org/aws/cp-psql-reader` via `tools/cp_psql.sh dev` — deferred (operator to grant); UI oracles are primary. - Dev UI: `me/jaah/debug/main/operator-login` and `org/console/dev-login` — verified. - AC7 dev: dev Jaah org accounts on two built-in providers — deferred (operator to confirm). Out of scope: changing account quota semantics or spillover; training a model from scratch; routing analysis personas before coder tasks have data; public marketplace billing; any ledger metric definition (owned by the outcome ledger epic). MUST NOT define, change or compute any ledger metric (success rate, survival, defects, cost per success, the survival window) — read them through 12243's interface. MUST NOT create `task_outcomes` or its writers. Done means: - AC3 Scoreboard router vs default survival and cost. → Verify: on dev as operator, open the operator Routing scoreboard and pick 30 days. Observe per-org rows with decisions count, agreement rate, and survival and cost per success for router-pick vs default, with intervals. As an org Admin of an org without the feature, observe no scoreboard link and a 403/404 on the route (D4). - AC7 Auto spreads across two allowed providers (dev part). → Verify: On dev after §6: on the Auto test project with two allowed providers, observe both providers in the scoreboard's dispatch breakdown and the exploration share ≤ the project knob. - AC12 Findings recorded (ledger part). → Verify: After §6, the ledger row for that coder task shows the security-findings facts. An API test asserts it. - AC14 Marketplace (numbers part). → Verify: The ledger numbers read "No ledger data yet" or "Insufficient data" before §6 and Wilson-interval figures after. - API tests: LedgerStats org-first then prior; prior carries no org identity; `routing_decision_id` set on the ledger row for a dispatched task; AC2 byte-identity matrix and AC7 simulation still pass. - Gates (AC16): `./dc check quick` green; changed-area suites green; merge lane `./dc check full` green.

Activity

  • Draft
  • Ready