Profile and cut hot endpoints

## Original requirement (verbatim) > "Make the control-plane API highly concurrent and distributed: many processes, no per-task connection caps." Epic 12693. Settled upstream: PRD (task 12694) and Architecture (task 12695). The operator overrode the PRD's multi-worker reading (D3): ONE Granian process per task on free-threaded CPython, async DB I/O on one shared pool. The full design (decisions D1–D9, risks, oracle tooling) is the `architecture` artifact of task 12695; read it (`artifact_read` / `task_read` on 12695) for the pinned seams S1–S5 (§4), the measured cause (§1), material risks (§6) and the burst-probe tooling and baseline (§8) before you start. This body carries this child's scope, acceptance and fences verbatim. ## Scope of this child: C7 — Hot-endpoint profiling (Architecture §5, verbatim) ### C7 — Hot-endpoint profiling - Use the existing CPU profiler (`app/cpu_profile.py`, operator setting `cpu_profiling`) on dev while loading the AC10 pages: Tasks board, task detail, Audit › Sessions, Machines. - Cut per-request cost on the 3–5 heaviest endpoints, starting with `/api/tasks?limit=50` (0.26 s serial): N+1 queries, over-fetching, and serialization. - **Bar:** remove the dominant cost the profile shows for each endpoint it touches, and report the before/after profile plus serial and burst timings (§8) as evidence. There is no numeric gate (PRD §6). The pass/fail bar for the epic stays AC10. - Works against whichever session API the file has when it starts: no dependency on C2/C3. Rebase resolves. ## Approach (planner) Profile first; fix only what the profile shows. Leave the profiler setting as it was after capture. ## Applies to every child (verbatim) Each child is one implementer's PR. **Dependencies are data/API only.** Children that edit the same file are not serialized; they rebase at merge. **Every child owns AC15:** its merge's Jaah Deploy must go green, and dev `/api/health` must answer 200. Every child that changes request-path behaviour also runs the S4 regression probe. ## Constraints (PRD §7, verbatim) ## 7. Constraints - Merges go through the merge lane on `./dc check full`. Each merge's Jaah Deploy must be watched to green (project instruction). - Migrations, if any, must be additive while two releases overlap (`docs/standards/coding-standards.md`). - The deploy overlap doubles tasks, so every connection-budget formula carries ×2. - Live state is read through the console UI only (CLAUDE.md hard rule). `tools/cp_psql.sh` was refused by the credential broker for this BA run (`org/aws/cp-psql-reader`), so current dev latency was not measured here. The Architect must get it (AS6, AC12). Credentials & reach: | Oracle | Credential / identity / environment | Status | |---|---|---| | Burst probe, AC3, AC5, AC8, AC10 (console as operator) | `me/jaah/debug/main/operator-login` on `app.dev.jaah.ai` | verified: `credential_get` via broker OK. Login 200 and timed GETs from this node (ts node of task 12695). | | AC12, C8 metrics (dev + main CloudWatch) | `me/jaah/system/bootstrap-admin` → `OrganizationAccountAccessRole` in `jaah-dev` / `jaah-main`, read-only | verified: ECS `describe-services` and CloudWatch `get-metric-statistics` succeeded on both from this node. | Out of scope: - Do NOT add a numeric load-test gate or a benchmark harness (PRD §6). - Do NOT convert routes to async (C3 batches); work against whichever session API the file has when you start. - Do NOT change task/RDS sizing. Epic-wide (PRD §6, verbatim): - **Do not** promote to `master` or change prod's live config as part of done (Q8). Prod tfvars may be edited only so that the R7 budget test covers them. Do not apply them to prod. - **Do not** remove or loosen anonymous/public abuse controls (AS2): the WAF, `SharedRateLimiter` public and trigger-intake limits, read-relay `MAX_SOCKETS_PER_IP` / `ANON_*`, site-chat per-IP limits, headscale edge nginx `limit_conn` / `limit_req` (`infra/jaah/headscale/nginx.conf`). - **Do not** remove resource-safety bounds (AS1): body, frame and response size limits, read timeouts, `pool_timeout`. - **Do not** remove per-ECS-task DB pool sizing as a concept, or exceed RDS `max_connections` (AS3). - **Do not** add a numeric load-test gate or a benchmark harness. None was asked for (Q4, Q7). - **Do not** change node-side caps that are not on the CP-call path: vnode conntrack limits (`apps/jaahd/vnodehelper/`), MicroVM egress `MAX_CONNECTIONS_PER_TAP`, reconcile probe parallelism, `pr_stats_capture` semaphore. - **Do not** change unrelated product limits: plan and org limits (`orgs.limit_*`), `max_accounts_per_node`, node session caps, worktrees-per-node. - **Do not** introduce a leader-elected singleton CP or pin `desired_count` to 1 (ADR-0142 anti-pattern still holds). - **Do not** add a process-local registry consulted by a request path (ADR-0142 anti-pattern still holds). - Do NOT change task CPU/memory, RDS class or autoscaling max (operator D2: sizing out of scope). - Do NOT reintroduce multi-worker / multiple processes per container (operator D3), and do NOT keep a uvicorn fallback (D8). Done means: - C7-a — Profiles captured on dev with `app/cpu_profile.py` (operator setting `cpu_profiling`) while loading the AC10 pages (Tasks board, task detail, Audit › Sessions, Machines); the 3–5 heaviest endpoints, starting with `/api/tasks?limit=50`, are named — oracle: report attaches before/after profile summaries. - C7-b — For each endpoint touched, the dominant cost the profile shows (N+1 queries, over-fetching, serialization) is removed, and the response is unchanged (existing route tests pass) — oracle: before/after profile + serial and §8 burst timings in the report; `./dc check full`. - C7-c (planner tightening) — each touched endpoint's serial latency on dev is lower after than before, and its query count per request is lower or equal — oracle: report table (before/after serial ms and query count per endpoint). - AC15 — Jaah Deploy to dev succeeds for this child's merge commit on `develop`, and dev `/api/health` answers 200 after it — oracle: GitHub Actions "Jaah Deploy" run for the merge commit is green; `curl https://dev.jaah.ai/api/health` → 200. (If the deploy is cancelled/superseded by another deploy, leave it and complete.)

Profiled dev; cut per-org/per-project fan-out in /api/tasks (84→36 stmts) and /api/projects (57→37); responses identical. PR #7754 merged (1b2bf7f) on full gate. Deploy superseded twice, so the dev after-timings are unmeasured. Memory: proposed 1 gotcha.