Retire sync DB, verify on dev

## Original requirement (verbatim) > "Make the control-plane API highly concurrent and distributed: many processes, no per-task connection caps." Epic 12693. Settled upstream: PRD (task 12694) and Architecture (task 12695). The operator overrode the PRD's multi-worker reading (D3): ONE Granian process per task on free-threaded CPython, async DB I/O on one shared pool. The full design (decisions D1–D9, risks, oracle tooling) is the `architecture` artifact of task 12695; read it (`artifact_read` / `task_read` on 12695) for the pinned seams S1–S5 (§4), the measured cause (§1), material risks (§6) and the burst-probe tooling and baseline (§8) before you start. This body carries this child's scope, acceptance and fences verbatim. ## Scope of this child: C8 — Retire sync DB + dev verification (Architecture §5, verbatim) ### C8 — Retire sync DB + dev verification - Delete the sync engines, `_LanedSession`, `SessionLocal`, `IdentitySessionLocal`, `background_lane` and `ReservedQueuePool`, `get_session`, the sync tfvars and env, and `apps/api/sync_routes_allowlist.txt`. `tools/check_async_routes.py` becomes absolute. - Final S4 budget: one async pool equal to the per-task total. - Run every remaining PRD oracle on dev (AC3–AC6, AC10–AC14) and record before/after: - the burst probe (§8); - prod-shape CloudWatch `DbPoolCheckedOutMax{Lane=async}` and ALB p99 on dev. - **C8 is not done until AC10 and AC11 pass on dev.** If a page or webhook still crosses 3 s, C8 fixes the endpoint itself, using the profiler as in C7. A follow-up may only add to that, never replace it. ## Approach (planner) Delete first, gate green, deploy, then run every oracle on dev; fix any endpoint still ≥3 s in this child. ## Applies to every child (verbatim) Each child is one implementer's PR. **Dependencies are data/API only.** Children that edit the same file are not serialized; they rebase at merge. **Every child owns AC15:** its merge's Jaah Deploy must go green, and dev `/api/health` must answer 200. Every child that changes request-path behaviour also runs the S4 regression probe. ## Constraints (PRD §7, verbatim) ## 7. Constraints - Merges go through the merge lane on `./dc check full`. Each merge's Jaah Deploy must be watched to green (project instruction). - Migrations, if any, must be additive while two releases overlap (`docs/standards/coding-standards.md`). - The deploy overlap doubles tasks, so every connection-budget formula carries ×2. - Live state is read through the console UI only (CLAUDE.md hard rule). `tools/cp_psql.sh` was refused by the credential broker for this BA run (`org/aws/cp-psql-reader`), so current dev latency was not measured here. The Architect must get it (AS6, AC12). Credentials & reach: | Oracle | Credential / identity / environment | Status | |---|---|---| | Burst probe, AC3, AC5, AC8, AC10 (console as operator) | `me/jaah/debug/main/operator-login` on `app.dev.jaah.ai` | verified: `credential_get` via broker OK. Login 200 and timed GETs from this node (ts node of task 12695). | | AC4, AC7 (Task action posting a PR comment; 3 runners on one node) | Same operator login, "Act as Gmind": project `mr-6382-static` (repo jaah-6382-backend), node ts2-node6 | verified: as the operator on dev, `GET /api/projects` lists `mr-6382-static` and `GET /api/nodes` lists `ts2-node6` (200, from this node). Launching runners is a console action on that project. **The AC4 comment check runs on a throwaway draft PR** that the builder opens on that repo for the test and closes afterwards, never on client work. | | AC6 and C1 "after an ECS replacement" (restart the CP mid-flight) | The deploy trigger: merging any PR to `develop` runs Jaah Deploy, which rolls every dev CP task. Watched with `jaah-forge` (task-authenticated) | verified: Jaah Deploy is the project's standard path (AC15). No console "restart CP" action exists, and no AWS write is used. | | AC9 (anonymous flood at the documented per-IP threshold) | No credential: anonymous requests to the read relay's anonymous handshake and a `SharedRateLimiter` public route (e.g. `/api/auth/login` with an unknown email, or trigger intake on `hooks.dev.jaah.ai`) | verified reach: anonymous HTTPS to `app.dev.jaah.ai` and `hooks.dev.jaah.ai` answers from this node (200/404, 0.11 s). The probe uses a throwaway email, never a real account, so no lockout. It runs in one short burst (seconds), because the per-IP bounds key on this node's egress IP, which other runners may share. | | AC11 (webhook / trigger timing) | Option A: GitHub App Recent Deliveries redelivery, credential `me/users/roman@gmind.tech/github_admin`. Option B: a dev trigger token minted in the console by the operator login (Automations › Triggers) and curl to `hooks.dev.jaah.ai` | verified: `github_admin` broker fetch OK (value not read). The operator login is verified above, and the trigger token is created through the console at test time. | | AC12, C8 metrics (dev + main CloudWatch) | `me/jaah/system/bootstrap-admin` → `OrganizationAccountAccessRole` in `jaah-dev` / `jaah-main`, read-only | verified: ECS `describe-services` and CloudWatch `get-metric-statistics` succeeded on both from this node. | | AC13, AC14 (gate) | Worktree only (`./dc check full`, `tools/check_context.py`) | verified: nothing beyond the worktree. | | AC15 (Jaah Deploy) | `jaah-forge` (task-authenticated) | verified: provided by the runner; no roster credential. | | Live DB reads (optional) | `me/jaah/system/cp-psql-reader` via `CP_AWS_CREDENTIAL=` with `tools/cp_psql.sh dev` | deferred: the credential fetch works, but the SSM tunnel refused the connection from this node. No oracle depends on it (CloudWatch covers AC12). | Out of scope: - Do NOT promote to `master` or apply prod config (PRD §6, Q8); main tfvars change only for the budget test. - Do NOT change DBOS's library-owned engine. Epic-wide (PRD §6, verbatim): - **Do not** promote to `master` or change prod's live config as part of done (Q8). Prod tfvars may be edited only so that the R7 budget test covers them. Do not apply them to prod. - **Do not** remove or loosen anonymous/public abuse controls (AS2): the WAF, `SharedRateLimiter` public and trigger-intake limits, read-relay `MAX_SOCKETS_PER_IP` / `ANON_*`, site-chat per-IP limits, headscale edge nginx `limit_conn` / `limit_req` (`infra/jaah/headscale/nginx.conf`). - **Do not** remove resource-safety bounds (AS1): body, frame and response size limits, read timeouts, `pool_timeout`. - **Do not** remove per-ECS-task DB pool sizing as a concept, or exceed RDS `max_connections` (AS3). - **Do not** add a numeric load-test gate or a benchmark harness. None was asked for (Q4, Q7). - **Do not** change node-side caps that are not on the CP-call path: vnode conntrack limits (`apps/jaahd/vnodehelper/`), MicroVM egress `MAX_CONNECTIONS_PER_TAP`, reconcile probe parallelism, `pr_stats_capture` semaphore. - **Do not** change unrelated product limits: plan and org limits (`orgs.limit_*`), `max_accounts_per_node`, node session caps, worktrees-per-node. - **Do not** introduce a leader-elected singleton CP or pin `desired_count` to 1 (ADR-0142 anti-pattern still holds). - **Do not** add a process-local registry consulted by a request path (ADR-0142 anti-pattern still holds). - Do NOT change task CPU/memory, RDS class or autoscaling max (operator D2: sizing out of scope). - Do NOT reintroduce multi-worker / multiple processes per container (operator D3), and do NOT keep a uvicorn fallback (D8). Done means: - C8-a — The sync engines, `_LanedSession`, `SessionLocal`, `IdentitySessionLocal`, `background_lane`, `ReservedQueuePool`, `get_session`, the sync tfvars/env and `apps/api/sync_routes_allowlist.txt` are deleted; `tools/check_async_routes.py` is absolute (zero sync handlers, zero sync-session imports app-wide) — oracle: `git grep` for each symbol returns nothing live; quick gate. - AC13 — `./dc check full` passes, including `infra/jaah/test_cp_autoscaling.py` with the final S4 budget (one async pool equal to the per-task total: main 27+4, dev 20+4), over dev and main tfvars with max tasks × 2 overlap — oracle: merge lane gate. - AC3 — | AC3 | Live console updates arrive exactly once on every CP task (cockpit_events fan-out). | Open the dev console Tasks board in two browser tabs, which the two dev CP tasks will likely serve from different tasks (re-read under operator D3: one process per task). Change a task's state in tab A. Tab B updates within 2 s, exactly once, with no duplicate rows or toasts. Repeat 10 times. | - AC4 — | AC4 | No double-posted forge comments under the async single-loop CP (spec R10 / issue #897 regression). | Run the existing anti-double-post tests in the gate. Also, on dev, trigger a Task action that posts a GitHub comment, double-clicking or sending two concurrent requests. Exactly one comment appears on the PR. | - AC5 — | AC5 | Node-bus commands reach nodes whichever CP task serves the request. | On dev, from the console, run a node command 20 times in a row, such as a worktree refresh or a node probe from Machines. All 20 succeed, with no "node not connected" error for a node whose heartbeat is live. | - AC6 — | AC6 | DBOS durable flows survive: unique executor per CP task, recovery intact. | Gate tests for `app/durable`. On dev, start a durable human-in-the-loop flow (an AskUserQuestion round), restart or redeploy the CP mid-flight, and answer. The answer is delivered and no workflow is stuck in PENDING (Audit or task view). | - AC10 — | AC10 | The ~3 s waits are gone on the console (R6). | On dev, open the console in Chrome with DevTools Network. Load the Tasks board, open 5 tasks, open Audit › Sessions, and open Machines, each 3 times. No `/api/*` XHR takes ≥3 s. Record the slowest timing per page in the report. Quality bar: pages render fully, not just their skeleton, within that time. | - AC11 — | AC11 | The ~3 s waits are gone on the public API / webhooks (R6). | On dev, send 20 trigger-intake or webhook requests (a forge webhook redelivery from the GitHub App's Recent Deliveries, or `curl` to a dev trigger URL with a dev token). Each response time is <3 s, as reported by GitHub delivery duration or curl `time_total`. | - AC12 — | AC12 | Root cause of the 3 s waits is identified and addressed (AS6). | The Architect's spec names the measured cause, with evidence from dev (pool_watch EMF metrics, ALB target response time, CPU). The build's report shows the same metric after the change no longer crosses 3 s. | - AC14 — `python3 tools/check_context.py` passes. ADR-0142 states the model: one Granian process per task, on free-threaded CPython, with async DB I/O. `git grep -niE "uvicorn" -- apps infra docs/policies docs/patterns docs/standards docs/adrs docs/architecture.md CLAUDE.md` returns nothing — oracle: run both, cite output. - C8-b — Before/after evidence recorded: the §8 burst probe and dev CloudWatch `DbPoolCheckedOutMax{Lane=async}` and ALB p99 — oracle: report table. - C8-c — C8 is not done until AC10 and AC11 pass on dev; a page or webhook still crossing 3 s is fixed here using the profiler (a follow-up may only add, never replace) — oracle: AC10/AC11 records. - AC15 — Jaah Deploy to dev succeeds for this child's merge commit on `develop`, and dev `/api/health` answers 200 after it — oracle: GitHub Actions "Jaah Deploy" run for the merge commit is green; `curl https://dev.jaah.ai/api/health` → 200. (If the deploy is cancelled/superseded by another deploy, leave it and complete.)