Keep cron run locks alive across RDS
Observed on dev, 2026-11-03 03:34:10–03:35:30 UTC. RDS rotated its managed master secret (`rds!db-…`, LastRotatedDate 03:35:11 UTC). For about 80 s, every cron firing on both CP tasks failed with `cron <id>: could not take the run lock`, which is `psycopg.OperationalError: … password authentication failed for user "jaah"`. The failure is raised from `app/cron_runtime.py` (RunLock.acquire → `db_credentials.psycopg_connect`). Over that window the CP logged 70 failed password authentications and 24 cron ticks ending `-> error`. `db_credentials` logged "re-resolving from DB_SECRET_ARN", but the re-resolved secret still held the old password until rotation finished. Every scheduled rotation will repeat this.
Ask: during a rotation, a cron's run-lock dial (and any other raw dialer) should retry briefly against the freshly resolved credential, or treat the auth refusal as a skipped tick rather than an error, so that no cron firing errors through a routine rotation.
Out of scope: rotation schedule, RDS configuration, pool sizing.
Done means:
- A test simulates the password being refused, then accepted after a re-resolve, and the cron tick does not end `-> error`.
- On dev, the next rotation's window shows 0 `ERROR cockpit-cron` lines in `/ecs/jaah-dev-cp` (CloudWatch Logs filter `"ERROR cockpit-cron"`).
Outcome
A cron run-lock dial refused during a credential rotation is now a credential_rotating skip with a warning, not an error. Refusals past 300 s are errors again. PR #7847 merged (303a761a63). Quick gate and tests green. The dev CloudWatch check is pending the next rotation. The deploy was superseded by a newer develop push.
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)