Async routes: org, billing, compliance

## Original requirement (verbatim) > "Make the control-plane API highly concurrent and distributed: many processes, no per-task connection caps." Epic 12693. Settled upstream: PRD (task 12694) and Architecture (task 12695). The operator overrode the PRD's multi-worker reading (D3): ONE Granian process per task on free-threaded CPython, async DB I/O on one shared pool. The full design (decisions D1–D9, risks, oracle tooling) is the `architecture` artifact of task 12695; read it (`artifact_read` / `task_read` on 12695) for the pinned seams S1–S5 (§4), the measured cause (§1), material risks (§6) and the burst-probe tooling and baseline (§8) before you start. This body carries this child's scope, acceptance and fences verbatim. ## Scope of this child: C3d — Routes: org, billing, compliance, signup (Architecture §5, verbatim) ### C3a–C3e — Route migration batches Same recipe for every batch: 1. Every handler in the fenced files becomes `async def` and takes `DbDep`. 2. Sync service calls go through `await session.run_sync(...)`, or are ported natively when local to the batch. 3. CPU-heavy work goes through `run_cpu` (response building over large row sets, crypto, archive/zip, markdown, JSON over MBs). 4. No transaction stays open across a node-bus or HTTP await (`release`). 5. The batch deletes its lines from `apps/api/sync_routes_allowlist.txt`. 6. The batch makes its fixed S4 shift of request-lane connections into the async pool. Each route file has exactly one owner: - `routes/org_shared_store.py` belongs to C3c only. C3d's `routes/org_*.py` glob excludes it. - `exec_dashboard/routes.py` belongs to C3d. - `console_relay/server.py` belongs to C3e. - **C3b owns AC4:** the anti-double-post gate tests (task actions under concurrent coroutines on one loop), plus the dev double-click check. Existing route tests must pass unchanged in behaviour. | Batch | Files (globs under `apps/api/app/`) | Approx. sync handlers | |---|---|---| | C3d org, billing, compliance, signup | `routes/orgs.py`, `routes/org_*.py`, `routes/billing*.py`, `routes/signup*.py`, `routes/compliance*.py`, `routes/customer*.py`, `routes/data_residency.py`, `routes/email_*.py`, `routes/hostings.py`, `routes/legal_acceptance.py`, `routes/scim.py`, `routes/plan_terms_check.py`, `routes/ce_*.py`, `routes/headscale_allowlist.py`, `routes/ip_allowlist.py`, `routes/trust_requests.py`, `routes/audit.py`, `routes/tenancy_exclusions.py`, `routes/prospect_data.py`, `operator_dashboard/routes.py`, `exec_dashboard/routes.py` (NOT `routes/org_shared_store.py`) | ~260 | Hot-path checks per batch: - C3a covers the runner heartbeat (`routes/agent/runner_liveness.py:285`), settle (`task_terminal.py:652`) and `console_runtime` (C3e). - C3b covers `wait_on_customer` (`tasks/transitions.py:460`) and answers (`tasks/answers.py:1229`). All of these hold a session across a wait today (recon). The batch must show they no longer do. ## Approach (planner) Mechanical per-file conversion using `await session.run_sync(...)` for shared sync services; port a service natively only when it is local to this batch. Start with the hot-path handlers named in scope. ## Applies to every child (verbatim) Each child is one implementer's PR. **Dependencies are data/API only.** Children that edit the same file are not serialized; they rebase at merge. **Every child owns AC15:** its merge's Jaah Deploy must go green, and dev `/api/health` must answer 200. Every child that changes request-path behaviour also runs the S4 regression probe. ## Constraints (PRD §7, verbatim) ## 7. Constraints - Merges go through the merge lane on `./dc check full`. Each merge's Jaah Deploy must be watched to green (project instruction). - Migrations, if any, must be additive while two releases overlap (`docs/standards/coding-standards.md`). - The deploy overlap doubles tasks, so every connection-budget formula carries ×2. - Live state is read through the console UI only (CLAUDE.md hard rule). `tools/cp_psql.sh` was refused by the credential broker for this BA run (`org/aws/cp-psql-reader`), so current dev latency was not measured here. The Architect must get it (AS6, AC12). Credentials & reach: | Oracle | Credential / identity / environment | Status | |---|---|---| | Burst probe, AC3, AC5, AC8, AC10 (console as operator) | `me/jaah/debug/main/operator-login` on `app.dev.jaah.ai` | verified: `credential_get` via broker OK. Login 200 and timed GETs from this node (ts node of task 12695). | | AC13, AC14 (gate) | Worktree only (`./dc check full`, `tools/check_context.py`) | verified: nothing beyond the worktree. | Out of scope: - Do NOT touch route files owned by another batch (each route file has exactly one owner, per the table). - Do NOT delete sync engines or the allowlist file (C8); do NOT convert background crons/pools (C4). - Do NOT touch `routes/org_shared_store.py` (owned by C3c). Epic-wide (PRD §6, verbatim): - **Do not** promote to `master` or change prod's live config as part of done (Q8). Prod tfvars may be edited only so that the R7 budget test covers them. Do not apply them to prod. - **Do not** remove or loosen anonymous/public abuse controls (AS2): the WAF, `SharedRateLimiter` public and trigger-intake limits, read-relay `MAX_SOCKETS_PER_IP` / `ANON_*`, site-chat per-IP limits, headscale edge nginx `limit_conn` / `limit_req` (`infra/jaah/headscale/nginx.conf`). - **Do not** remove resource-safety bounds (AS1): body, frame and response size limits, read timeouts, `pool_timeout`. - **Do not** remove per-ECS-task DB pool sizing as a concept, or exceed RDS `max_connections` (AS3). - **Do not** add a numeric load-test gate or a benchmark harness. None was asked for (Q4, Q7). - **Do not** change node-side caps that are not on the CP-call path: vnode conntrack limits (`apps/jaahd/vnodehelper/`), MicroVM egress `MAX_CONNECTIONS_PER_TAP`, reconcile probe parallelism, `pr_stats_capture` semaphore. - **Do not** change unrelated product limits: plan and org limits (`orgs.limit_*`), `max_accounts_per_node`, node session caps, worktrees-per-node. - **Do not** introduce a leader-elected singleton CP or pin `desired_count` to 1 (ADR-0142 anti-pattern still holds). - **Do not** add a process-local registry consulted by a request path (ADR-0142 anti-pattern still holds). - Do NOT change task CPU/memory, RDS class or autoscaling max (operator D2: sizing out of scope). - Do NOT reintroduce multi-worker / multiple processes per container (operator D3), and do NOT keep a uvicorn fallback (D8). Done means: - C3d-a — Every handler in this batch's files is `async def` taking `DbDep`; this batch's lines are deleted from `apps/api/sync_routes_allowlist.txt`; `tools/check_async_routes.py` passes — oracle: quick gate + `git diff` of the allowlist. - C3d-b — Existing route tests pass unchanged in behaviour — oracle: `./dc check full` via the merge lane. - C3d-c — No converted handler holds a transaction across a node-bus/HTTP/sleep await, and none calls sync I/O (`requests`, a sync session) on the loop; the batch's hot-path holders named in scope are shown fixed — oracle: the C2 held-across-await test helper exercised on those handlers; PR body cites file:line per holder. - C3d-d — CPU-heavy response building in this batch goes through `run_cpu` — oracle: PR body lists each offloaded call file:line. - AC13 — `./dc check full` passes, including `infra/jaah/test_cp_autoscaling.py` with this batch's fixed S4 shift applied to both tfvars: the sync request lane (`cp_db_sync_pool_size` first, then `cp_db_sync_max_overflow`, as C2 renames them) goes down and `cp_db_async_pool_size` goes up by the same amount. C2's effective-env test passes for both tfvars, so a lane drained to 0 is disabled and never boots at its code default — oracle: merge lane gate. - S4 regression check — after this child deploys, the §8 burst probe on dev is no worse than the baseline at N=24 (`/api/tasks` N=24 max 4.6 s) — oracle: run the burst probe, record wall/p50/max in the PR body and report. Evidence, not a numeric gate; a child that makes it worse is not done. - AC15 — Jaah Deploy to dev succeeds for this child's merge commit on `develop`, and dev `/api/health` answers 200 after it — oracle: GitHub Actions "Jaah Deploy" run for the merge commit is green; `curl https://dev.jaah.ai/api/health` → 200. (If the deploy is cancelled/superseded by another deploy, leave it and complete.)