Post-deploy regression detection and revert

Task 12274 · Queued · Coder · created · updated

## Requirement (verbatim) 5. "Regression detection: post-deploy error-rate and latency deltas from the adapters against the previous release baseline; a breach files a task and, where the project policy says so, a revert task." Settled: D8 "Pull the monitors' APIs after a production deploy, using an org credential: Sentry stats, Datadog metrics, CloudWatch metrics. PagerDuty supplies incidents only." D12 revert policy per project off/draft/auto-file; "A revert is always a task through the normal PR and merge lane. Jaah never merges a revert outside the lane." A3 defaults: error rate ≥ 1.5× baseline with ≥ 20 events, or p95 ≥ 1.3×, windows 60 min. A4 baseline = same-length window before the current production deploy. Architect A-7: revert = a CP filing attributed to the regression automation and its owner. Design: architecture artifact of Task 12249, §8 (windows, pulls, breach, resolve, not_measured), §1.2 A-7/A-9, §6.6 (revert, moved here from the incident→task child), §5.3 `metrics` shape. ## Scope (architecture C7 + revert §6.6) - Credential types `sentry` (`base_url`, `org_slug`, `auth_token`) and `datadog` (`site`, `api_key`, `app_key`) in `credentials/registry.py` (app-side, no migration); CloudWatch uses `aws`. Adapter `metrics` config validation references a credential by id. - `regression.schedule`: production push anchor → `regression_due_at = finished_at + regression_window_min`, `pending`. Cron job `regression_due`: per due deployment × adapter trigger with `metrics`, post window `[finished_at, +W]`, baseline `[started_at − W, started_at]`, cut at the next anchor (≥15 min else `not_measured: superseded by deploy #N`). Sentry `events-stats` (pin the endpoint with a recorded fixture first; doc-unverified), Datadog `POST /api/v2/query/timeseries`, CloudWatch `GetMetricData` metric math. Always store `deployment_metrics`; breach → `incidents.record(type='regression', fingerprint=regression:{env}:{metric}, severity='error', deployment_id, facts={metric, baseline, post, delta_abs, delta_pct, window})`; a later measured non-breach resolves it; missing credential / failed pull (two retries 5 min apart) → `not_measured: <reason>`, nothing filed. - Revert in `events.post_file` hook: for a `regression` incident when `revert_policy != 'off'`: one coder task `Revert: <PRs>`, body = confirmed suspects else `high` ones + "revert through a normal PR", `state = ready` if auto_file else `draft`, `started_by_id` = automation owner, `created_by_automation_id`, claim `role='revert'`. No regression automation → no revert; incident shows "Revert: no automation". No code path merges. Credentials & reach: - AC-21 live regression — a CloudWatch-read AWS credential — **deferred** — `org/aws` probed: `sts get-caller-identity` succeeds (IAM user jaah-node-launcher), `cloudwatch:ListMetrics` AccessDenied. Operator R5 will grant CloudWatch read (suggested `org/aws-cloudwatch-qa`, GetMetricData/ListMetrics/PutMetricData on a sandbox namespace); re-verify with `credential_get` + `sts get-caller-identity` + `list-metrics` before the live run; until then fixture tests - Sentry/Datadog pulls — no `sentry`/`datadog` credential — **deferred** — operator R4: fixture tests with vendor sample payloads - UI checks on dev — `org/console/dev-login` (org Admin), `https://dev.jaah.ai` — credential **verified**; environment **deferred** for this node (operator R4: run from a tailnet node) - AC-29 gates — worktree only — **verified** Out of scope: Do NOT build an APM or metric store (only the two-window pulls and `deployment_metrics`). Do NOT merge a revert outside the merge lane in any policy mode (D12); no merge call anywhere. Do NOT add a migration. Do NOT store a monitor credential anywhere but the existing credential store. Epic out of scope (verbatim): "building an APM or log store; replacing the customer's monitor; auto-merging a revert without the project's policy; any scoring or ranking logic (the outcome ledger epic reads the states this epic writes)." Done means: - AC-21: "after a production deploy on a project with a Sentry, Datadog or CloudWatch adapter and a configured org API credential, Jaah pulls the error rate (and p95 latency where the monitor offers it) for the post-deploy window and for the baseline window. It stores both and the delta, visible on the Deployments row detail. A breach writes a `regression` incident and, with the feature ON and the project opted in, files a task whose body states the metric, baseline, post-deploy value, delta (absolute and %), window and suspect PRs. No breach: nothing is filed, and the delta is still stored. Missing credential or failed pull: the row says 'not measured: <reason>', and nothing is filed." — fixture-driven tests per monitor (breach, no breach, missing credential, failed pull, superseded window); live CloudWatch run deferred until the R5 credential exists (name it in the PR). - AC-22: "`off` files no revert task. `draft` files a revert task as `draft`. `auto-file` files it `ready`. In every mode the revert goes through a normal PR and the merge lane, and no code path merges a revert directly." — one test per setting; grep oracle over the diff: no merge call outside the lane. - AC-26 (OFF check): with `production_qa` OFF, `deployment_metrics` and the regression incident are still written and nothing is filed — test. - AC-29: `./dc check quick` green; changed-area tests green; merge lane full gate green.

Activity

  • Draft
  • Ready