Alarm on starved identity and background DB pools

Task 7144 · Done · Coder · created · updated

Task 7119 publishes per-lane EMF metrics from every control-plane process (`apps/api/app/metrics_emf.py::pool_lines`, dimension `Lane` = request | identity | background): `DbPoolCheckedOutMax`, `DbPoolOverflowMax`, `DbPoolCheckoutTimeouts`, and on the request lane `DbReleaseRoundTripsMsP50`. Nothing alarms on them yet. Ask: add CloudWatch alarms in `infra/jaah/terraform/monitoring.tf` (main only, the paging topic) for: 1. `DbPoolCheckoutTimeouts` Sum > 0 for 3 consecutive minutes on Lane=identity (each one means a 500/503 on an auth read). 2. The same on Lane=background (dispatch stalls). Tune the thresholds on dev's figures first (the namespace is set on dev too). Extend `tests/test_metrics_emf.py` / `infra/jaah/test_monitoring.py` so the alarm names stay pinned to the emitter's constants. Out of scope: changing pool sizes; the `/api/health` verdict. Done means: - `terraform plan` for main shows the two alarms and nothing else. - The monitoring contract tests pass. - `./dc check quick` is green.

Outcome

Added main-only paging alarms on DbPoolCheckoutTimeouts Lane=identity/background (Sum>0, 3 consecutive min) plus contract tests; main plan diff = exactly the 2 alarms; quick gate green; #5818 merged, develop deploy green. Draft follow-up #7172 (gate tmp sweep race).

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)