Production QA loop deployment and incident management

Task 12247 · Running · Epic · created · updated

QA stops at the merge lane's post-merge tree check (fleet-task-lifecycle.md §Post-merge gate verification); forge CI is a live read-only view (forge_ci.py), Jaah Deploy's result never reaches the CP, CloudWatch alarms only page Slack (alarm_relay.py), and the only condition-to-task path is in-process cron alarms (cron_alarms.py). Close the loop to production. PRD: the attached proposal report (Task 12195), decisions table and "Epic 2". 1. Deploy ingestion: consume forge deployment_status / workflow_run events on the existing `github` trigger source (and each supported forge's equivalent) into a `deployments` table (project, environment, status, commit range, started/finished) mapped to the merged PRs it contains. Writes the ledger's merged → deployed edge and starts its survival clock. First consumer: Jaah Deploy on the Jaah project, replacing the runner charter's manual deploy watch. 2. Monitor adapters (bring your own): new trigger source kinds `sentry`, `datadog`, `pagerduty`, `cloudwatch` (HTTP ingress with signature verification, beside triggers.py SOURCE_KINDS) normalised into an `incidents` table: fingerprint, first/last seen, count, severity, environment, stack frames, release/commit when sent. One row per fingerprint; recurrences update, never duplicate. Adapters count against the org's trigger quota. 3. Suspect linkage: the monitor's release/commit when present; else deploys in the first-seen window narrowed by stack-frame paths against their diffs. Store confidence, show it, let an operator correct it. Writes the ledger's incident_linked state on the origin outcomes. 4. Incident → task: automation When kinds `incident`, `deploy_failed`, `health_breach` (automations.py When kinds) with the existing `task` action; files a `triage` or `coder` task carrying the incident body and suspect links; dedup per fingerprint; per-project severity threshold, quiet hours and auto-file vs draft. The fix PR references the incident; the incident closing after the fix's deploy marks the fixer survived and the origin incident_linked. 5. Regression detection: post-deploy error-rate and latency deltas from the adapters against the previous release baseline; a breach files a task and, where the project policy says so, a revert task. Store merge-lane stage timings per PR (already in gate logs) and file a task on a sustained slowdown of a named suite. 6. Native probes (later phase): project-declared health URLs and smoke commands run from the project's own node right after a deploy, writing the same incidents table. 7. Measurement is always collected (deployments, incidents, linkage, regression deltas, the ledger edges). Every behaviour change — filing a task, filing a revert, running probes — is behind the plan feature `production_qa` with org override and per-project opt-in. Dogfood on the Jaah project first. Out of scope: building an APM or log store; replacing the customer's monitor; auto-merging a revert without the project's policy; any scoring or ranking logic (the outcome ledger epic reads the states this epic writes). Done means: - A merged PR on a connected project shows its deploy and status in the console within one minute of the forge event, and its ledger row carries the deployed edge. - A Sentry-shaped (and a generic webhook-shaped) incident on a test project creates one incidents row and, with production_qa ON and the project opted in, one linked task; a second occurrence updates the row and files nothing; with the feature OFF the incidents row is still written and no task is filed. - Closing the incident after the fix's deploy flips the fixer's outcome to survived and the origin's to incident_linked. - A failed Jaah Deploy files a task on the Jaah project with no runner watching it. - A post-deploy error-rate breach against the previous release files a task with the delta in its body. - Required checks: ./dc check quick green; changed-area tests green; merge lane full gate green.

Subtasks

Attachments

Activity

  • Created
  • Updated
  • Runner started
  • Ready
  • Ready
  • Ready
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response