Keep cron run locks alive across RDS

Task 13073 · Done · Coder · created · updated

Observed on dev, 2026-11-03 03:34:10–03:35:30 UTC. RDS rotated its managed master secret (`rds!db-…`, LastRotatedDate 03:35:11 UTC). For about 80 s, every cron firing on both CP tasks failed with `cron <id>: could not take the run lock`, which is `psycopg.OperationalError: … password authentication failed for user "jaah"`. The failure is raised from `app/cron_runtime.py` (RunLock.acquire → `db_credentials.psycopg_connect`). Over that window the CP logged 70 failed password authentications and 24 cron ticks ending `-> error`. `db_credentials` logged "re-resolving from DB_SECRET_ARN", but the re-resolved secret still held the old password until rotation finished. Every scheduled rotation will repeat this. Ask: during a rotation, a cron's run-lock dial (and any other raw dialer) should retry briefly against the freshly resolved credential, or treat the auth refusal as a skipped tick rather than an error, so that no cron firing errors through a routine rotation. Out of scope: rotation schedule, RDS configuration, pool sizing. Done means: - A test simulates the password being refused, then accepted after a re-resolve, and the cron tick does not end `-> error`. - On dev, the next rotation's window shows 0 `ERROR cockpit-cron` lines in `/ecs/jaah-dev-cp` (CloudWatch Logs filter `"ERROR cockpit-cron"`).

Outcome

A cron run-lock dial refused during a credential rotation is now a credential_rotating skip with a warning, not an error. Refusals past 300 s are errors again. PR #7847 merged (303a761a63). Quick gate and tests green. The dev CloudWatch check is pending the next rotation. The deploy was superseded by a newer develop push.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)