Keep cron ticks alive when a request burst
Reproduction / observed: in Task 12744's dev verification, a 60-way parallel GET burst as the operator filled the single async pool (dev 20+4). Requests waited the 10 s pool_timeout. 226 tracebacks followed, all reading `QueuePool limit of size 20 overflow 4 reached`. Crons that ticked during the burst failed their tick the same way. The cause was the burst, not the crons. Before the epic, background work had its own lane (11 connections). Task 12743 (PR #7800) moved it onto the shared request pool through `run_on_loop` (`apps/api/app/db_async.py:459`), so one user's burst can starve the scheduler: task-pipeline, lease-drain and dispatch crons. My own N=24 probe on 2026-11-03 hit the 24-connection ceiling (CloudWatch `Jaah/dev DbPoolCheckedOutMax{Lane=async}` = 24) with all requests answered 200, so a burst does not need to be large.
Cause: request-path checkouts and background checkouts compete on equal terms for the same QueuePool (`apps/api/app/db_async.py:106-111`). Nothing keeps headroom for background work.
Fix (keep ONE pool, per operator decision "One async pool per task"): admit request-path sessions (`get_db`, `apps/api/app/db_async.py:274`) through a per-process asyncio semaphore sized pool_size + max_overflow − a small background reserve (e.g. 4, as one constant beside `db_pool_sizes`). Background checkouts through `run_on_loop` / the bridge bypass it. A burst then queues requests on the semaphore, and cron ticks still get connections. Do not add a second engine or lane, and do not change pool sizes.
Acceptance:
- Unit test: with the request semaphore exhausted, a background `run_on_loop` session still checks out within 1 s. Oracle: the new test in `apps/api/tests/`. It fails with the semaphore removed.
- On dev after deploy, a 60-way burst of `/api/tasks?limit=50` (the burst probe, Architecture §8 of task 12695) produces zero cron tick failures in `/ecs/jaah-dev-cp` during the burst window. Oracle: a CloudWatch Logs filter on the cron failure lines.
- `infra/jaah/test_cp_autoscaling.py` is unchanged and still green.
Out of scope: pool sizing, task CPU/memory, RDS class (D2); restoring separate lanes; anonymous abuse limits; route conversion.
Done means:
- The test and the dev burst evidence above are recorded in the report.
- `./dc check full` is green via the merge lane, and Jaah Deploy is green with dev `/api/health` returning 200.
Outcome
Request admission keeps 4 pool connections for background work. #7824 merged 8e7afc8e via the full gate; Jaah Deploy green, dev health 200. Dev 60-way burst: 0 cron failures (was 43). Filed draft #13073 (RDS rotation fails cron locks). Memory: none applied.
Artifacts
Activity
- Ready
- Runner started
- Runner ended (done)