Request auth reads from their own DB pool
Root-cause follow-up to the main CP pool outage (Task 6722, 2026-10-17 19:04–19:12 EDT). #5733 adds the safety net: a starved pool fails `/api/health`, and the task self-exits on ECS. This task removes the cause.
**What happened.** One CP task's app pool (15 + 15) stayed saturated for 7+ minutes and logged 3,000+ `QueuePool limit … reached` errors, while RDS sat idle (CPU ~20%, DB load 2).
- **Trigger:** 5 runner launches in 9s.
- **Where checkouts timed out:**
| Frame | Timeouts |
|---|---|
| `node_credentials._node_for_token` | 1076 |
| `authz._resolve_principal` (opens its own `SessionLocal()`) | 608 |
| `auth._session_user` | 252 |
| cron_runtime | ~280 |
- **Inferred mechanism:** hold-and-wait. A request holds its route session's connection, then needs a second for auth or principal resolution (`app/db.py` already notes "a cookie-authenticated object route takes three"). The request threadpool is sized to the pool's capacity (`config.request_thread_limit`). Lease-drain's dispatch workers, cron threads, `asyncio.to_thread` callers and DBOS also draw on the same pool, outside that bound. So 30 threads can each hold one connection and wait on another, and each frees only after `pool_timeout` (10s).
- **Not proven:** the holders' stacks were never captured, because py-spy fails `Permission Denied` on main. #5733 adds the in-process dump. **First step: read the next `pool_dump` on main, or reproduce under load locally, to confirm the holders before building.**
**Requirements**
1. Auth, principal and node-token reads (`_node_for_token`, `_resolve_principal`, `_session_user`, and any other per-request identity read) check out from a dedicated small engine/pool, so they can never wait on connections held by route sessions. Alternatively, prove they always reuse the request's own session; pick the one the measurements support.
2. Background users get a bounded share of the pool, or their own pool, so a launch burst cannot starve requests: lease-drain dispatch workers (`lease_drain.py` executor), cron firings, `asyncio.to_thread` DB callers.
3. The per-environment connection budget (`DB_POOL_SIZE`/`DB_MAX_OVERFLOW`, `DBOS_POOL_SIZE`, `infra/jaah/test_cp_autoscaling.py`) is updated for the new pool(s) against each RDS class's `max_connections` × max task count, including deploy overlap.
4. A load test, local or on dev, reproduces the starvation on today's code (a burst of node-token and cookie requests plus N concurrent lease-drain dispatches) and shows it gone after the change.
Out of scope: the `/api/health` starvation verdict and self-exit (#5733); the WAF limits; merge lane logic.
Done means:
- The reproduction is red before and green after, cited in the report.
- `tests/test_cp_autoscaling` and the connection budget tests pass for every environment.
- `./dc check quick` is green; the lane merges the PR; Jaah Deploy succeeds.
Outcome
Auth reads now use their own DB pool, and background work uses a separate bounded pool. Route connections are released before thread hops, and the budget is rebalanced (main 744/760). Load test: develop 72/72 failed, this branch 210/210 OK. Quick gate and full suite green. PR #5786 merged; its deploy run was superseded by the next merge. Follow-up draft #7119.
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)