Background work on the async loop

## Original requirement (verbatim) > "Make the control-plane API highly concurrent and distributed: many processes, no per-task connection caps." Epic 12693. Settled upstream: PRD (task 12694) and Architecture (task 12695). The operator overrode the PRD's multi-worker reading (D3): ONE Granian process per task on free-threaded CPython, async DB I/O on one shared pool. The full design (decisions D1–D9, risks, oracle tooling) is the `architecture` artifact of task 12695; read it (`artifact_read` / `task_read` on 12695) for the pinned seams S1–S5 (§4), the measured cause (§1), material risks (§6) and the burst-probe tooling and baseline (§8) before you start. This body carries this child's scope, acceptance and fences verbatim. ## Scope of this child: C4 — Background async (Architecture §5, verbatim) ### C4 — Background async - Convert to asyncio tasks on the serving loop, using `async_session()`: - cron firings: `app/cron_runtime.py:297,1193,1199`, keeping `RunLock` (a session advisory lock on its own async connection); - lease-drain's per-node workers (`app/lease_drain.py:2708`, 8); - trigger intake (`app/trigger_intake.py:194,581`); - node-bus side pools (`app/routes/node_bus.py:109,111`); - `forge_ci.py:124`, `project_memory/search.py:55`, `cp_peers.py:359`; - per-call pools in `forge_repos.py`, `git_connections.py`, `routes/git_connect.py`, `instance_type_browse.py`, `acp_registry_icons.py`, `routes/commands.py:1627`; - raw threads in `automations.py:405`, `credentials/testers.py:548`, `durable/nodevault.py:508,1056`. - Concurrency that was a thread-pool size becomes an `asyncio.Semaphore` only where it protects an external rate (forge API). Otherwise it is removed. - The background lane moves into the async pool (the S4 C4 shift). The identity lane already moved in C2. - Code still on a foreign thread uses `run_on_loop` (S1). A DBOS step is converted to a native `async` step where DBOS allows it. - DBOS steps that open app sessions use `async_session()`. DBOS's own engine stays. - `_AnswerLock` and the cron lock keep their dedicated advisory-lock connection: a raw dialer, counted in `RAW_DIALERS_PER_TASK`. ## Approach (planner) Convert the hottest background holder first (lease-drain, cron firings), then the per-call pools; prefer native async DBOS steps. ## Applies to every child (verbatim) Each child is one implementer's PR. **Dependencies are data/API only.** Children that edit the same file are not serialized; they rebase at merge. **Every child owns AC15:** its merge's Jaah Deploy must go green, and dev `/api/health` must answer 200. Every child that changes request-path behaviour also runs the S4 regression probe. ## Constraints (PRD §7, verbatim) ## 7. Constraints - Merges go through the merge lane on `./dc check full`. Each merge's Jaah Deploy must be watched to green (project instruction). - Migrations, if any, must be additive while two releases overlap (`docs/standards/coding-standards.md`). - The deploy overlap doubles tasks, so every connection-budget formula carries ×2. - Live state is read through the console UI only (CLAUDE.md hard rule). `tools/cp_psql.sh` was refused by the credential broker for this BA run (`org/aws/cp-psql-reader`), so current dev latency was not measured here. The Architect must get it (AS6, AC12). Credentials & reach: | Oracle | Credential / identity / environment | Status | |---|---|---| | Burst probe, AC3, AC5, AC8, AC10 (console as operator) | `me/jaah/debug/main/operator-login` on `app.dev.jaah.ai` | verified: `credential_get` via broker OK. Login 200 and timed GETs from this node (ts node of task 12695). | | AC13, AC14 (gate) | Worktree only (`./dc check full`, `tools/check_context.py`) | verified: nothing beyond the worktree. | Out of scope: - Do NOT change DBOS's own library-owned engine. - Do NOT convert route handlers (C3a–C3e). - Do NOT touch `pr_stats_capture` semaphore or reconcile probe parallelism (PRD §6). Epic-wide (PRD §6, verbatim): - **Do not** promote to `master` or change prod's live config as part of done (Q8). Prod tfvars may be edited only so that the R7 budget test covers them. Do not apply them to prod. - **Do not** remove or loosen anonymous/public abuse controls (AS2): the WAF, `SharedRateLimiter` public and trigger-intake limits, read-relay `MAX_SOCKETS_PER_IP` / `ANON_*`, site-chat per-IP limits, headscale edge nginx `limit_conn` / `limit_req` (`infra/jaah/headscale/nginx.conf`). - **Do not** remove resource-safety bounds (AS1): body, frame and response size limits, read timeouts, `pool_timeout`. - **Do not** remove per-ECS-task DB pool sizing as a concept, or exceed RDS `max_connections` (AS3). - **Do not** add a numeric load-test gate or a benchmark harness. None was asked for (Q4, Q7). - **Do not** change node-side caps that are not on the CP-call path: vnode conntrack limits (`apps/jaahd/vnodehelper/`), MicroVM egress `MAX_CONNECTIONS_PER_TAP`, reconcile probe parallelism, `pr_stats_capture` semaphore. - **Do not** change unrelated product limits: plan and org limits (`orgs.limit_*`), `max_accounts_per_node`, node session caps, worktrees-per-node. - **Do not** introduce a leader-elected singleton CP or pin `desired_count` to 1 (ADR-0142 anti-pattern still holds). - **Do not** add a process-local registry consulted by a request path (ADR-0142 anti-pattern still holds). - Do NOT change task CPU/memory, RDS class or autoscaling max (operator D2: sizing out of scope). - Do NOT reintroduce multi-worker / multiple processes per container (operator D3), and do NOT keep a uvicorn fallback (D8). Done means: - C4-a — Every item listed in scope runs as an asyncio task on the serving loop using `async_session()` (or `run_on_loop` for a remaining foreign thread), each cited file:line in the PR body; thread-pool sizes are removed or become an `asyncio.Semaphore` only where they protect an external (forge API) rate — oracle: PR body table + tests. - C4-c (critic MAJOR, drained background lane) — No live `background_executor`/`background_lane` caller remains, including `pool_watch.py:535` and `node_credentials.py:408`, which are outside the listed scope. A gate test boots the app with the background lane at 0 and exercises the converted paths — oracle: `git grep -nE "background_executor|background_lane" -- apps/api/app` shows only definitions in `db.py` (C8 deletes them); the test passes. - C4-b — `RunLock`, `_AnswerLock` and the cron lock keep their dedicated advisory-lock connection, counted in `RAW_DIALERS_PER_TASK` — oracle: existing lock tests pass; `test_cp_autoscaling.py` counts them. - AC6 — | AC6 | DBOS durable flows survive: unique executor per CP task, recovery intact. | Gate tests for `app/durable`. On dev, start a durable human-in-the-loop flow (an AskUserQuestion round), restart or redeploy the CP mid-flight, and answer. The answer is delivered and no workflow is stuck in PENDING (Audit or task view). | - AC13 — `./dc check full` passes, including `infra/jaah/test_cp_autoscaling.py` with the C4 shift (background lane: main 11, dev 8 into the async pool) — oracle: merge lane gate. - S4 regression check — after this child deploys, the §8 burst probe on dev is no worse than the baseline at N=24 (`/api/tasks` N=24 max 4.6 s) — oracle: run the burst probe, record wall/p50/max in the PR body and report. Evidence, not a numeric gate; a child that makes it worse is not done. - AC15 — Jaah Deploy to dev succeeds for this child's merge commit on `develop`, and dev `/api/health` answers 200 after it — oracle: GitHub Actions "Jaah Deploy" run for the merge commit is green; `curl https://dev.jaah.ai/api/health` → 200. (If the deploy is cancelled/superseded by another deploy, leave it and complete.)