Backfill outcome ledger history

Task 12281 · Queued · Coder · created · updated

Epic 12243 (plan task 12246). Full settled inputs: the PRD (Task 12244) and the Architecture (Task 12245) of this epic — read them with `task_read`/`artifact_read` on those tasks; section numbers below (R…/AC…/§…) cite them. Where PRD and the original proposal (Task 12195) differ, the PRD governs. Depends on the ledger core child (writer API, tables). ## Original requirement (verbatim) > "1. Ledger: one `task_outcomes` row per build attempt that settles or fails. Attribution: agent (built-in provider or SDK agent + commit), model family and concrete model, account auth_kind, persona, review tier, routing decision id (nullable, future), project, org, node class. Timeline: ready → first active → PR opened → enqueued → merged → deployed → survival window end. Gate facts: lane verdicts per push, dispatch_attempts, pushes after enqueue, non-runner commits before merge, security findings from review lanes / CI scans. Cost: priced tokens over all attempts (failed included), node minutes, lane minutes." PRD requirement carried (verbatim): R1.7 "A one-off backfill creates rows for every task settled before release (D11). The rows carry merge, lane and cost facts, `backfilled = true`, and null for anything that cannot be rebuilt." D11: "**All history, counted.** Every task settled before release gets backfilled rows, flagged `backfilled = true` and included in metrics. A fact that cannot be rebuilt stays null and is excluded from that metric only." Epic out-of-scope: "re-scoring tasks older than a one-off backfill of merge, lane and cost facts." ## Approach (settled — realize, do not reopen) Architecture §13. - One-off **durable** job in `apps/api/app/durable/` (restart resumes it): `outcome_ledger/backfill.py`, idempotent keyed by (task, dispatch_seq), batches of 500 tasks with one commit per batch (no long lock on `tasks`). - Started from an operator action **Backfill outcome ledger** on the operator settings page (UI-only rule; no SQL/API script), with progress/last-run shown there. - Per settled task: one row per `agent_sessions` attempt ordered by `started_at` (dispatch_seq, model, tokens, node, account); merge/diff/files from `task_prs`/`task_pr_stats`/`task_pr_changes`; lane verdicts per head and `enqueued_at` from `merge_lane_rows` (via `writer.add_head`/`set_head_verdict`); cost from `Task.metrics` per session. A task with no sessions → one row from the task alone. Unknowns null (`review_tier='unknown'`, `nonrunner_commits`, `auth_kind` if the account is gone, `node_*`). Survival state from `merged_at + window`. Writes go through `outcome_ledger.writer` only. - Cutoff: a task is backfilled only if it has zero ledger rows and settled before the ledger-core child's deploy timestamp (stored as the job's cutoff); tasks with live rows are skipped, so `dispatch_seq` never collides. - Never re-runs LLM scoring, never reopens a task, never re-scans origins outside their window. Credentials & reach: - pytest ACs: none beyond the worktree (`./dc` stack) — verified (worktree). - Dev UI flows as Jaah-org Owner and as operator: `me/jaah/debug/main/operator-login` (operator, switches into tenants); environment `https://app.dev.jaah.ai` (public dev console; the internal `dev.jaah.ai` ALB did not answer from h2 within 15 s) — verified: credential_get ok; `app.dev.jaah.ai` → HTTP 200 from h2. - Live dev DB reads: `tools/cp_psql.sh dev` → broker refused `org/aws/cp-psql-reader` — not needed by any oracle (every AC is a UI flow or a test). Out of scope: the Outcome panel "Backfilled" marker UI (child "Build task Outcome panel" owns AC1.5's browser check); any change to tasks or their states. Out of scope (epic, verbatim): "any routing behaviour (the routing epic reads this ledger); new production signal sources (the production QA epic writes deployed/incident_linked); a public index or tenant-to-tenant sharing; re-scoring tasks older than a one-off backfill of merge, lane and cost facts." Do NOT touch (PRD A8): no change to `apps/api/app/agent_resolution.py` resolution order, `apps/api/app/jaah_scheduler.py` account placement, or `apps/api/app/jaah_routing/` (`routing_decision_id` stays null); no deploy ingestion, monitor adapters, incidents table or new trigger source kinds (`apps/api/app/triggers.py` untouched; `deployed_at`/`incident_linked` are columns/states nothing writes); no public page, no unauthenticated endpoint, no tenant-to-tenant view, no tenant name on any operator-plane aggregate; no retired or rewritten weekly outcome-review job (`apps/api/app/operator_dashboard/review.py`, `apps/api/app/operator_dashboard/quality.py`, D4); backfill writes merge, lane and cost facts only, never re-runs LLM scoring, never re-opens old tasks; benchmark mode never merges a shadow PR and never changes the origin task's dispatch. No CP forge call (ADR-0156): forge facts come only through the `outcome_scan` runner. Migrations additive only (new tables + nullable/defaulted columns). Done means: - R1.7 test: a fixture of pre-release tasks (merged via lane, self-merged, failed, no sessions, account deleted) produces the expected rows with `backfilled = true`, merge/lane/cost facts present, unknowns null (never zero/epoch); running the job twice creates no duplicates; a task that already has live rows is skipped (mixed-case test); killing it mid-run and resuming finishes with the same row set. `pytest apps/api -k backfill` green. - The job leaves `tasks` rows, task states and LLM review rows unchanged (asserted in the test). - Dev: as the operator on `https://app.dev.jaah.ai`, run **Backfill outcome ledger** from operator settings and observe it complete with a row count; re-run and observe 0 new rows. - AC1.5 (data half): "after the backfill on dev, open a task settled before release. Its Outcome panel shows a \"Backfilled\" marker, with merge and cost facts present and unknowns shown as \"—\"." — the panel itself is verified by the Outcome-panel child; this child must leave dev backfilled so that check can run. - Gate (epic Done means, verbatim: "Required checks: ./dc check quick green; changed-area tests green; merge lane full gate green.") — `./dc check quick` green; changed-area pytest/vitest green; merge lane `./dc check full` green; Jaah Deploy of the PR succeeds.

Activity

  • Draft
  • Ready