Alarm on starved identity and background DB pools
Task 7119 publishes per-lane EMF metrics from every control-plane process (`apps/api/app/metrics_emf.py::pool_lines`, dimension `Lane` = request | identity | background): `DbPoolCheckedOutMax`, `DbPoolOverflowMax`, `DbPoolCheckoutTimeouts`, and on the request lane `DbReleaseRoundTripsMsP50`. Nothing alarms on them yet.
Ask: add CloudWatch alarms in `infra/jaah/terraform/monitoring.tf` (main only, the paging topic) for:
1. `DbPoolCheckoutTimeouts` Sum > 0 for 3 consecutive minutes on Lane=identity (each one means a 500/503 on an auth read).
2. The same on Lane=background (dispatch stalls).
Tune the thresholds on dev's figures first (the namespace is set on dev too). Extend `tests/test_metrics_emf.py` / `infra/jaah/test_monitoring.py` so the alarm names stay pinned to the emitter's constants.
Out of scope: changing pool sizes; the `/api/health` verdict.
Done means:
- `terraform plan` for main shows the two alarms and nothing else.
- The monitoring contract tests pass.
- `./dc check quick` is green.
Outcome
Added main-only paging alarms on DbPoolCheckoutTimeouts Lane=identity/background (Sum>0, 3 consecutive min) plus contract tests; main plan diff = exactly the 2 alarms; quick gate green; #5818 merged, develop deploy green. Draft follow-up #7172 (gate tmp sweep race).
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)