Request auth reads from their own DB pool

Task 6983 · Done · Coder · created · updated

Root-cause follow-up to the main CP pool outage (Task 6722, 2026-10-17 19:04–19:12 EDT). #5733 adds the safety net: a starved pool fails `/api/health`, and the task self-exits on ECS. This task removes the cause. **What happened.** One CP task's app pool (15 + 15) stayed saturated for 7+ minutes and logged 3,000+ `QueuePool limit … reached` errors, while RDS sat idle (CPU ~20%, DB load 2). - **Trigger:** 5 runner launches in 9s. - **Where checkouts timed out:** | Frame | Timeouts | |---|---| | `node_credentials._node_for_token` | 1076 | | `authz._resolve_principal` (opens its own `SessionLocal()`) | 608 | | `auth._session_user` | 252 | | cron_runtime | ~280 | - **Inferred mechanism:** hold-and-wait. A request holds its route session's connection, then needs a second for auth or principal resolution (`app/db.py` already notes "a cookie-authenticated object route takes three"). The request threadpool is sized to the pool's capacity (`config.request_thread_limit`). Lease-drain's dispatch workers, cron threads, `asyncio.to_thread` callers and DBOS also draw on the same pool, outside that bound. So 30 threads can each hold one connection and wait on another, and each frees only after `pool_timeout` (10s). - **Not proven:** the holders' stacks were never captured, because py-spy fails `Permission Denied` on main. #5733 adds the in-process dump. **First step: read the next `pool_dump` on main, or reproduce under load locally, to confirm the holders before building.** **Requirements** 1. Auth, principal and node-token reads (`_node_for_token`, `_resolve_principal`, `_session_user`, and any other per-request identity read) check out from a dedicated small engine/pool, so they can never wait on connections held by route sessions. Alternatively, prove they always reuse the request's own session; pick the one the measurements support. 2. Background users get a bounded share of the pool, or their own pool, so a launch burst cannot starve requests: lease-drain dispatch workers (`lease_drain.py` executor), cron firings, `asyncio.to_thread` DB callers. 3. The per-environment connection budget (`DB_POOL_SIZE`/`DB_MAX_OVERFLOW`, `DBOS_POOL_SIZE`, `infra/jaah/test_cp_autoscaling.py`) is updated for the new pool(s) against each RDS class's `max_connections` × max task count, including deploy overlap. 4. A load test, local or on dev, reproduces the starvation on today's code (a burst of node-token and cookie requests plus N concurrent lease-drain dispatches) and shows it gone after the change. Out of scope: the `/api/health` starvation verdict and self-exit (#5733); the WAF limits; merge lane logic. Done means: - The reproduction is red before and green after, cited in the report. - `tests/test_cp_autoscaling` and the connection budget tests pass for every environment. - `./dc check quick` is green; the lane merges the PR; Jaah Deploy succeeds.

Outcome

Auth reads now use their own DB pool, and background work uses a separate bounded pool. Route connections are released before thread hops, and the budget is rebalanced (main 744/760). Load test: develop 72/72 failed, this branch 210/210 OK. Quick gate and full suite green. PR #5786 merged; its deploy run was superseded by the next merge. Follow-up draft #7119.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)