Capture security findings in ledger
Epic 12243 (plan task 12246). Full settled inputs: the PRD (Task 12244) and the Architecture (Task 12245) of this epic — read them with `task_read`/`artifact_read` on those tasks; section numbers below (R…/AC…/§…) cite them. Where PRD and the original proposal (Task 12195) differ, the PRD governs. Builds on the ledger writer, the outcome scan, the metrics engine and the Performance page.
## Original requirement (verbatim)
> "1. Ledger: one `task_outcomes` row per build attempt that settles or fails. Attribution: agent (built-in provider or SDK agent + commit), model family and concrete model, account auth_kind, persona, review tier, routing decision id (nullable, future), project, org, node class. Timeline: ready → first active → PR opened → enqueued → merged → deployed → survival window end. Gate facts: lane verdicts per push, dispatch_attempts, pushes after enqueue, non-runner commits before merge, security findings from review lanes / CI scans. Cost: priced tokens over all attempts (failed included), node minutes, lane minutes."
> "4. Metrics per configuration (agent × model family × persona × review tier) with roll-ups per agent, model, project, org: success rate (survived ÷ settled+failed, operator-cancelled excluded); first-pass acceptance (merged with zero lane failures, zero re-dispatch, zero human fix commits ÷ merged); defects introduced per 100 merges; rework (lane failures + re-dispatches + post-enqueue pushes + human fix commits per merged task, plus rework hours); completion time ready→merged median/p90 and agentic time; cost per successful task (Σ cost of all attempts ÷ survived); security issues per 100 tasks, credited to the detecting lane and debited to the authoring configuration. Wilson 95% intervals; a configuration ranks only at ≥ 20 outcomes, else \"insufficient data\"; stratify by persona and a complexity bucket (diff size, files touched, repo language, clarity score)."
PRD (verbatim): D14 "**Runner review lanes and CI scans.** Lanes report findings (severity, file, detecting lane) on the settle call. CI security-scan results are read from the PR's forge checks." Phase 3 (PRD §5): "Security findings: lanes report on settle, plus CI scans read from forge checks (D14). This adds the security metric." Constraint (PRD §6): "Phase 3 adds a settle-call field. Runner prompts that describe the settle call must name it (`docs/policies/ai-code-policy.md`, prompt text as executable)."
## Approach (settled — realize, do not reopen)
Architecture §3.4, §11.
- One additive migration: `outcome_findings` (+ org-transfer rule); `task_outcomes.security_findings` kept in step by `writer.end_row(..., findings=)`.
- `SettleIn`/fail body gain optional `findings: list[{severity, path?, rule?, title, detector}]` (≤ 200; Any-typed, stored `malformed` rather than 422 — the `pr_stats` leniency rule); tool schemas in `apps/jaahd/agent_vocabulary.py`; prompt sweep: review-lane text in `docs/policies/code-review-policy.md` §lanes, `dispatch_compose/role_prose.py`, `portfolio/fleet/runner-charter.md` name it.
- CI scans: the outcome scan's per-sha checks matching `security_check_patterns` (default `codeql|semgrep|snyk|trivy|gitleaks|bandit|dependency-review|sonar|checkmarx|osv` + per-project additive list in project settings) become `ci` findings (severity from annotations where given, else `medium` for a failing check).
- Metric #7 security_per_100 fed from findings, debited to the authoring configuration, with the separate `by_detector` credit list; a **Security / 100** column on Command center › Performance.
Credentials & reach:
- pytest ACs: none beyond the worktree (`./dc` stack) — verified (worktree).
- Dev UI flows as Jaah-org Owner and as operator: `me/jaah/debug/main/operator-login` (operator, switches into tenants); environment `https://app.dev.jaah.ai` (public dev console; the internal `dev.jaah.ai` ALB did not answer from h2 within 15 s) — verified: credential_get ok; `app.dev.jaah.ai` → HTTP 200 from h2.
- Live dev DB reads: `tools/cp_psql.sh dev` → broker refused `org/aws/cp-psql-reader` — not needed by any oracle (every AC is a UI flow or a test).
Out of scope: any other settle field; routing; benchmark.
Out of scope (epic, verbatim): "any routing behaviour (the routing epic reads this ledger); new production signal sources (the production QA epic writes deployed/incident_linked); a public index or tenant-to-tenant sharing; re-scoring tasks older than a one-off backfill of merge, lane and cost facts."
Do NOT touch (PRD A8): no change to `apps/api/app/agent_resolution.py` resolution order, `apps/api/app/jaah_scheduler.py` account placement, or `apps/api/app/jaah_routing/` (`routing_decision_id` stays null); no deploy ingestion, monitor adapters, incidents table or new trigger source kinds (`apps/api/app/triggers.py` untouched; `deployed_at`/`incident_linked` are columns/states nothing writes); no public page, no unauthenticated endpoint, no tenant-to-tenant view, no tenant name on any operator-plane aggregate; no retired or rewritten weekly outcome-review job (`apps/api/app/operator_dashboard/review.py`, `apps/api/app/operator_dashboard/quality.py`, D4); backfill writes merge, lane and cost facts only, never re-runs LLM scoring, never re-opens old tasks; benchmark mode never merges a shadow PR and never changes the origin task's dispatch. No CP forge call (ADR-0156): forge facts come only through the `outcome_scan` runner. Migrations additive only (new tables + nullable/defaulted columns).
Done means:
- The AC4.1 fixture is extended with findings: "a fixture of known rows gives hand-computed values for all seven metrics and their Wilson bounds (to 3 dp). Run `pytest apps/api -k outcome_metrics` and observe it pass." — security_per_100 and `by_detector` hand-computed.
- A settle with findings stores them and sets `security_findings`; malformed findings are stored `malformed`, never a 422; a test asserts AC1.3 still holds (no other outcome field is runner-writable).
- A scan report with a failing `codeql` check on an origin's merge sha yields one `ci` finding with severity `medium`; a project-added pattern is honoured.
- Every prompt that describes `task_complete`/`task_fail` names `findings` (a test or grep-based check in the prompt tests).
- Dev: Command center › Performance shows the Security / 100 column with interval and n, readable at 390 px.
- Gate (epic Done means, verbatim: "Required checks: ./dc check quick green; changed-area tests green; merge lane full gate green.") — `./dc check quick` green; changed-area pytest/vitest green; merge lane `./dc check full` green; Jaah Deploy of the PR succeeds.
Activity
- Draft
- Ready