Evidence collector framework, DB/GitHub collectors

Part of epic 6213. Child C2 of 9. Depends on C1 (`packages/evidence` + `compliance_evidence/store.file`), C4 (`trust.md`, whose Type II line this child flips) and C5 (its migration — one alembic migration per wave; this child's migration chains after C5's). ## Original requirement (verbatim) > "Identify which security and compliance certifications Jaah's competitors have — SOC 2, HIPAA, ISO 27001, etc. — and close the gap. Without breaking anything :)" **R2 — SOC 2 Type II readiness: in-house recurring evidence (D-4)** (verbatim): - Every SOC 2 check in the readiness checklist that an agent can run unattended runs on a schedule as a Jaah system cron and files a dated artifact to the evidence store through `tools/evidence_store.py` (existing folders only, or a folder added there). - A check that fails or drifts raises the existing alarm/paging path — never silent. - `R-SOC-6` and D7 updated in place: recurring evidence is in-house; Drata is no longer the plan. - Checks that cannot run unattended (need an owner session/credential) are listed with why. - The same collectors serve ISO 27001/42001 evidence where controls overlap (no duplicate collectors). ## Scope (architecture §2.3, §2.4, §3.2) **Contract** `apps/api/app/compliance_evidence/collectors.py`: ```python @dataclass(frozen=True) class Finding: status: Literal["pass", "drift"] summary: str # one line, human, no secrets — goes into the cron run detail body: bytes # JSON, sorted keys, UTF-8 filename: str # "<collector-id>-<YYYY-MM-DD>.json" @dataclass(frozen=True) class Collector: id: str; label: str # e.g. "Evidence: CloudWatch alarms" control: str # a packages.evidence.CONTROLS member criteria: tuple[str, ...]; cron: str # 6-field, daily, staggered 04:00–05:30 UTC run: Callable[[Session, datetime], Finding] # raises on collector ERROR REGISTRY: tuple[Collector, ...] ``` Registration is LITERAL: one `_system(EVIDENCE_<ID>, …)` call per collector in `apps/api/app/cron_system_activities.py` `_register_all` (~line 757); each activity type is a kebab-case module constant (`EVIDENCE_SECURITY_ARCHIVE = "evidence-security-archive"` …) — the tenancy census (`apps/api/test_tenancy_plane_sites.py:1624-1628`) refuses computed types, so NO loop over REGISTRY. Each call passes a thin invoke delegating to shared `run_collector(collector, scheduled_at, context)`. A test asserts REGISTRY ids, constants and `_system` calls agree, and every REGISTRY id has a seeded row. Outcomes: | outcome | artifact | alarm | cron run | |---|---|---|---| | pass | filed | `resolve_alarm("compliance-evidence:<id>")` | `acted`; detail = summary + `s3://` URI | | drift | filed | `raise_alarm(dedup_key="compliance-evidence:<id>", …)` naming the drift | `acted`; detail names drift + URI | | exception / write failed | none | `raise_alarm` with error class+message (no secrets) | `failed` | `cron_alarms._own_key` gains prefix `COMPLIANCE_EVIDENCE_KEY_PREFIX = "compliance-evidence:"`. Run detail MUST carry the `s3://` URI (how AC4 is observed without AWS creds). **Seeding ON**: new alembic migration (timestamp id) inserting one `crons` row per C2 collector, `is_system=true`, **`enabled=true`**, same `INSERT … ON CONFLICT (activity_type) WHERE is_system DO NOTHING` shape as `_SYSTEM_CRONS` in `202610130700_baseline.py:1681-1701`. No flag, no env knob. **down_revision = the head at build time (C5's migration)**; C3 chains after this one. **Test edits**: `apps/api/tests/test_cron_system_activities.py` `_EXPECTED` (~line 34) and `_SEEDED_ENABLED` (~line 89) — ADD the new rows only; no pre-existing row removed or altered. **Six collectors**: | id | source | folder | criteria | drift | |---|---|---|---|---| | `security-archive` | S3 list of archive bucket (role already has it) | `control-checks` | CC2.1, CC7.2 | a day missing in last 30 | | `retention-sweep` | DB `cron_runs` for `retention_sweep` | `offboarding` | CC6.5, C1.2 | no `acted` firing in 26 h | | `access-events` | DB `activity_events` counts (role_binding_*, login_failed, credential_read, 7 d) | `control-checks` | CC6.2, CC7.3, C1.1 | never (record only) | | `mfa-posture` | DB `orgs.mfa_skip` per org | `control-checks` | CC6.1 | never; lists orgs with mfa_skip on and cites row `R-MFA-SKIP` (no register copy in code) | | `github-settings` | GitHub REST: the rulesets/repo settings `tools/github_settings.py export` reads, ported into `app/` (same JSON shape); declared state from `infra/github/rulesets/*.json` baked into the image (add to `infra/local/Dockerfile.api` COPY if absent) | `github-settings` | CC5.2, CC7.1, CC8.1 | ruleset drift vs `infra/github/` | | `dependabot` | GitHub REST open Dependabot alerts with age | `control-checks` | CC6.8 | an alert past the vulnerability SLA in the information-security policy | **GitHub credential**: new operator setting `compliance_github_credential_id` in a new "Compliance" group of `app/operator_settings.py`, modelled on `pr_review_reviewer_credential_id`. While unset, both GitHub collectors take the **error leg** with alarm "GitHub credential not set: Settings > Compliance" (visible, never silent). Add owner action **I-15** (set it on prod; needs a prod console session, `R-SOC-18`) to the readiness checklist I-table. **Docs**: rewrite `R-SOC-6` in `docs/compliance/risk-register.md` (line ~91) IN PLACE: evidence is collected in-house by the collectors, citing `apps/api/app/compliance_evidence/`; drop Drata (D7 exists only as that row's citation — updating the row updates D7). Add a readiness-checklist sub-section mapping collector id → criterion, plus the not-unattended list with reasons: entitlement export (management account, cross-account assume); aggregated Security Hub (security account); `cp_readonly_check.sh` (proves a different DB role cannot write — the api runs as writer); DR restore test (destructive cost action, owner's schedule under H-1/A1.3); MFA behavioural probe (needs I-6 session); endpoint checks (I-8, H-22); every signature and attestation. Flip `docs/compliance/public/trust.md`'s line "SOC 2 Type II — planned" to "SOC 2 Type II — evidence collection running" (pointer → the new readiness-checklist section; no unfinished marker). Credentials & reach: AC4, AC5: Crons page, trigger collector, see run detail with `s3://` key and alarm Task | `jaah/debug/main/operator-login` (operator login, valid on dev and main) / operator / dev CP `app.dev.jaah.ai` | **verified**: credential fetch succeeded; `GET https://app.dev.jaah.ai/api/public/settings` answered 200 from `h2`. AC4 pass/drift legs of the `github-settings` and `dependabot` collectors on dev | a dev-held GitHub credential with jaah-ai admin and Dependabot read / dev CP | **deferred**: no such credential is verified on dev; copying a prod org-admin token into dev is not warranted (architect decision). Those legs are proven by HTTP-fake unit tests; the dev error leg ("credential not set" alarm) is proven live; prod is configured by owner action I-15. AC6 | worktree-only | **verified**. AC-X2 | runner's own forge login via `jaah-forge` | **verified**. Out of scope: Do NOT write the AWS collectors or IAM (C3). Do NOT run entitlement export, DR restore, cp_readonly_check or MFA probe unattended. Do NOT copy a prod GitHub token into dev. Do NOT alter existing crons' rows, schedules or behaviour. Epic-wide (PRD §6, verbatim): "**Do not** obtain, buy, sign or contract anything (cert body, auditor, Drata, SafeBase, pentest firm, NDAs); **do not** contact outside parties. **Do not** claim any certification as held. **Do not** change HIPAA mode behaviour. **Do not** flip the **prod** legal publish switch or promote to `master`. **Do not** widen the audit boundary to dev; **do not** alter existing controls' behaviour (MFA, sessions, retention, IAM) to satisfy ISO. **Do not** put instance data (people, account ids) in reusable docs. **Do not** add a new evidence-store folder outside `tools/evidence_store.py`'s list." Done means: - AC4 (R2) Each unattended-runnable readiness check runs on its schedule on the **dev** control plane and a dated artifact appears in the evidence store per run. States: pass → artifact filed; fail/drift → artifact filed AND alarm fires; collector error → alarm fires (never silent). → verify: on dev console Crons page, trigger a collector, observe a success journal entry and the new evidence key; force a failing check (e.g. a test fixture or a known-failing dev setting) and observe the alarm. (For the 4 DB/S3 collectors live on dev; for the 2 GitHub collectors: pass/drift (deferred — no dev GitHub credential; proven by HTTP-fake unit tests), error leg live.) - AC5 (R2) The collectors are ON by default in the real deployment (registered system crons, enabled), not behind a flag left off. → verify: dev console lists them enabled with a next-run time. - AC6 (R2) `R-SOC-6` text no longer defers to Drata; D7 updated in place. → verify: read the row. - AC11 for the flipped Type II line: no status claims a certificate not issued → cross-check vs readiness checklist. - Registry↔constants↔`_system`↔seed agreement test passes; `alembic heads` shows exactly one head. - AC-X1 (A3) Nothing breaks: `./dc check full` green on every PR; existing legal pages, sign-up links, HIPAA mode and existing crons behave unchanged. → verify: gate output; existing legal-page and cron tests pass unchanged. - AC-X2 (A3) Every PR's Jaah Deploy succeeds on dev. → verify: GitHub deploy run status.

6 SOC 2 evidence collectors as enabled daily crons (#5393, #5394, #5395 merged; full gate green; dev deploys ok). Live on dev: 4 pass with s3:// keys, 2 GitHub fail with "credential not set". Alarm Task unverified on dev: no cto project there (follow-up).