Move node-wire operator reads to async pool

Task 12963 · Done · Coder · created · updated

After C3a (Task 12738, PR #7796), the S4 burst probe on dev regressed for `GET /api/nodes` at N=24. Max went from 3.4 s (same-day baseline) to ~5.4 s across three runs, while p50 improved (2.5 s → 1.4–2.2 s). `/api/tasks` improved: its max is 2.3–3.8 s against a 4.6 s baseline. Cause: `list_nodes`, `list_nodes_page` and `get_node` (`apps/api/app/routes/nodes/reads.py`) build their rows through the node wire (`routes/nodes/node_wire.py::_node_out` and helpers). Those helpers call operator-plane reads that open their own sync sessions: - `node_allocation.jaah_cloud` / `placement` / `locked_for` - `vnodes.host_of` / `disk_usage_of` / `fleet_session` - `hosted_machines.hosted_among` The handlers therefore run whole in `db_async.run_blocking_in_session`, on the sync request lane. After the fixed S4 shifts, dev's sync request lane is a single connection (`cp_db_sync_pool_size = 1`, `cp_db_sync_max_overflow = 0`), so a burst serialises on it. Main has 1+1. Ask: move those operator-plane reads onto the async pool, e.g. an async fleet session as `routes/nodes/delete.py::_fleet_session` does, or reads that take a session. Then `list_nodes`, `list_nodes_page`, `get_node` (and `nodes/update.py`, `nodes/metrics.py`, `nodes/register.py`, which use the same wire) can drop `run_blocking_in_session` and run on `DbDep` via `run_sync`. Keep tenancy exactly as today: operator-plane reads stay detached plus `operator_scope`, and are censused in `apps/api/test_tenancy_scope_sites.py`. Out of scope: changing the S4 lane sizes or the per-task connection budget (sizing is out of scope for epic 12693); other route batches' hatch users; the `operator_settings` cache-miss read. Done means: - `GET /api/nodes`, `/api/nodes/page` and `/api/nodes/{id}` no longer call `run_blocking_in_session`, and no sync session opens on the loop. A test wraps them in `tests/async_db_helpers.py::no_connection_across_awaits()` and asserts no `SessionLocal` checkout on the loop thread. - The §8 burst probe on dev, after deploy: `/api/nodes` N=24 max is no worse than 3.4 s. Record wall/p50/max. - `./dc check full` is green via the merge lane, and Jaah Deploy is green.

Outcome

Node GETs (/api/nodes, /page, /{id}, /metrics) now run on DbDep; operator-plane reads lend the route connection to an async-pool session. #7853 merged, full gate + Jaah Deploy green; dev /api/nodes N=24 max 1.6–2.3 s (target ≤3.4). Memory: none applied.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)