Incident state machine and recorder frame
## Epic context (verbatim)
"QA stops at the merge lane's post-merge tree check (...); the only condition-to-task path is in-process cron alarms (cron_alarms.py). Close the loop to production."
Requirement 2 (verbatim, the rule this child implements): "... normalised into an `incidents` table: fingerprint, first/last seen, count, severity, environment, stack frames, release/commit when sent. One row per fingerprint; recurrences update, never duplicate."
Requirement 7 (verbatim): "Measurement is always collected (deployments, incidents, linkage, regression deltas, the ledger edges). Every behaviour change — filing a task, filing a revert, running probes — is behind the plan feature `production_qa` with org override and per-project opt-in."
Design: architecture artifact of Task 12249, §0, §1.2 A-2/A-6/A-12/A-14/A-17, §4 (`incidents.record`, `Observation`, `Recorded`, `recorder.handles/record`, `cron.register`), fingerprints, delivery reasons, §6.3 step 3 (held plumbing), §13 risks 1, 6, 7, 10.
## Scope (architecture C1, core half)
- `incidents.record(session, project_id, obs)`: INSERT … ON CONFLICT (project_id, fingerprint) DO NOTHING then SELECT … FOR UPDATE; stale (`occurred_at < last_transition_at`) → `ignored` + `ignored_stale` event; resolve on unknown fingerprint → ignored, no row; count absolute → max, increment → +1; occurrence on resolved → `reopened` (raise suppressed inside 15-min cooldown, A-14); calls `linkage.compute`/`linkage.on_resolved` stubs; returns `RaiseIntent` per §4 incl. the A-17 30-min throttle on `occurred`; writes `incident_events`. Severity scale `debug<info<warning<error<critical`. `fingerprints.py` per §4.
- Recorder frame at the very top of `trigger_intake._Processor._handle`: when `recorder.handles(trigger,row)`, run `recorder.record` inside `session.begin_nested()`; failure rolls back the savepoint, logs, settles `failed` reason `record_error`; success settles `filtered` reason `recorded` (excluded from `filings_in_last_hour`) unless the trigger has an automation and is not `record_only`. Intents queued on the processor and flushed via `events.flush` AFTER the row's commit. `record_only: true` (github `source_config`) and every adapter kind make `_automation` and `automation_service.sync_trigger` refuse automation adoption (A-2; today an automation-less trigger adopts one and would file a task per event).
- Delivery state `held` plumbing: excluded from `_next_row`, `triggers_with_work` and `filings_in_last_hour`; `not_before` honoured. Human labels for every delivery reason in the run log ("Recorded", "Production QA off", "Project not opted in", "Held: quiet hours until …").
- `production-qa` system activity (`cron_system_activities`, 60 s) + `cron.register(name, fn)`: each registered job in its own transaction, caught and logged per job.
Credentials & reach:
- AC-29 gates — worktree only — **verified** (nothing external)
Out of scope: Do NOT add source kinds, adapters, deploy handlers, linkage logic, or task filing (sibling children). Do NOT change `alarm_relay.py`'s Slack paging (A6). Do NOT add an org trigger quota (D6). Epic out of scope (verbatim): "building an APM or log store; replacing the customer's monitor; auto-merging a revert without the project's policy; any scoring or ranking logic (the outcome ledger epic reads the states this epic writes)."
Done means:
- AC-10 (unit): "a second delivery with the same fingerprint updates `last_seen` and `count` and creates no new row. A resolve event (or the vendor's resolved state) moves it to `resolved`. A later occurrence moves it to `reopened` on the same row." — unit tests: two occurrences → one row, count 2; resolve → resolved; occurrence → reopened, same id, events history opened/occurred/resolved/reopened; concurrent duplicate insert (two sessions) → one row; stale event ignored; absolute vs increment counts.
- A-17/A-14: occurred storm raises at most once per 30 min without a claim; reopen inside 15 min of resolve raises nothing — unit tests.
- Recorder frame: a recorder exception leaves the delivery `failed`/`record_error` and no partial rows (savepoint test); a `record_only` github trigger with no automation adopts NO automation and files no task (regression test on `_automation`/`sync_trigger`).
- `held` rows are not picked by `_next_row` and not counted in the hourly cap — tests.
- AC-29: `./dc check quick` green; changed-area tests green; merge lane full gate green.
Activity
- Draft
- Ready
- Runner started