Free-threaded Python and Granian runtime

## Original requirement (verbatim) > "Make the control-plane API highly concurrent and distributed: many processes, no per-task connection caps." Epic 12693. Settled upstream: PRD (task 12694) and Architecture (task 12695). The operator overrode the PRD's multi-worker reading (D3): ONE Granian process per task on free-threaded CPython, async DB I/O on one shared pool. The full design (decisions D1–D9, risks, oracle tooling) is the `architecture` artifact of task 12695; read it (`artifact_read` / `task_read` on 12695) for the pinned seams S1–S5 (§4), the measured cause (§1), material risks (§6) and the burst-probe tooling and baseline (§8) before you start. This body carries this child's scope, acceptance and fences verbatim. ## Scope of this child: C1 — Runtime: free-threaded Python + Granian (Architecture §5, verbatim) ### C1 — Runtime: free-threaded Python + Granian - **Image** (`infra/local/Dockerfile.api`): - `apk add python-3.15t` (Wolfi ships it: 3.15.2_git20261030). Keep the same image for the console-relay service. - Build from source in the `deps` stage the packages that have no free-threaded wheel: `psycopg[c]` (libpq-dev + compiler) and `bcrypt` on 3.15t (needs Rust). - Drop `uvicorn[standard]` and add `granian` (2.8.x, cp315t wheel). - **Fallback rule:** if the full dependency set cannot import with `sys._is_gil_enabled() is False` on 3.15t, switch to `python-3.14t` and record the failing package in the PR body. - **Launcher:** S5 in `apps/api/entrypoint.sh`, and in `infra/jaah/terraform/modules/ecs/main.tf:1328` (the ECS command) and the local compose `command:`. - **Proxy headers (correctness, must ship):** Granian does not rewrite `request.client` from `X-Forwarded-For`. `app/routes/public.py::check_rate_limit` keys on `request.client.host`, so without a fix every client would share one global bucket (a public-login lockout of everyone; see the entrypoint comment). Add Starlette/ASGI proxy-header middleware that trusts only `127.0.0.1` (nginx), placed outermost in `create_app`. Test: two different `X-Forwarded-For` values get separate buckets. - **WebSocket keepalive:** uvicorn pinged every 20 s (`--ws-ping-interval 20`, mirrors `apps/jaahd/node_bus.py`). If Granian has no server-side ping option, add an app-level ping every 20 s, with the same 20 s timeout, in the node-bus and console-relay WS handlers, so a half-open socket is still detected on schedule. Test it. - **Free-threading race audit:** module-level mutable state touched from several threads (the `threading.Lock` users; dicts/sets mutated from `background_executor` threads) is reviewed for check-then-act races that the GIL used to mask. Fix with locks. The full suite runs on the free-threaded interpreter in the gate. - The audit explicitly covers the R2 process-local items and the lifespan actors (`app/main.py:210-244`), each recorded as safe or fixed: - `cron_runtime._fire_seen` - `node_send_intent._in_flight` - `support_agent.assist._live` - `db._held` - `user_activity`'s in-process locks - `ip_allowlist` caches - `tenancy/exclusion_log`'s dict - `pool_watch`, task notifier, Slack socket, live cast, cloud workers - **Docs (R8):** - ADR-0142 is edited in place: one process per task, served by Granian on free-threaded CPython. - Update the launcher comments in `apps/api/entrypoint.sh`, `infra/local/Dockerfile.api:191-196`, `apps/api/app/worker_guard.py` and `modules/ecs/variables.tf:148-164`, plus `infra/local/README.md` §prodlike. All describe the present; uvicorn is not mentioned. - **Acceptance:** - Dev task logs show one Granian process and the `free-threaded OK … GIL off` line, after deploy and after an ECS replacement. - The proxy-header test passes, and so does the WS keepalive test. - The quick and full gates are green on the new interpreter. - Jaah Deploy is green and dev `/api/health` returns 200. - AC5 (20 node commands) and AC3 (two-tab live updates) still pass on dev after this child alone. ## Approach (planner) Land the image/interpreter change and the launcher together so the dev deploy never runs a half state. Run the free-threaded import check first (cheapest failure point), then the full suite on it, then deploy and soak. ## Applies to every child (verbatim) Each child is one implementer's PR. **Dependencies are data/API only.** Children that edit the same file are not serialized; they rebase at merge. **Every child owns AC15:** its merge's Jaah Deploy must go green, and dev `/api/health` must answer 200. Every child that changes request-path behaviour also runs the S4 regression probe. ## Constraints (PRD §7, verbatim) ## 7. Constraints - Merges go through the merge lane on `./dc check full`. Each merge's Jaah Deploy must be watched to green (project instruction). - Migrations, if any, must be additive while two releases overlap (`docs/standards/coding-standards.md`). - The deploy overlap doubles tasks, so every connection-budget formula carries ×2. - Live state is read through the console UI only (CLAUDE.md hard rule). `tools/cp_psql.sh` was refused by the credential broker for this BA run (`org/aws/cp-psql-reader`), so current dev latency was not measured here. The Architect must get it (AS6, AC12). Credentials & reach: | Oracle | Credential / identity / environment | Status | |---|---|---| | Burst probe, AC3, AC5, AC8, AC10 (console as operator) | `me/jaah/debug/main/operator-login` on `app.dev.jaah.ai` | verified: `credential_get` via broker OK. Login 200 and timed GETs from this node (ts node of task 12695). | | AC6 and C1 "after an ECS replacement" (restart the CP mid-flight) | The deploy trigger: merging any PR to `develop` runs Jaah Deploy, which rolls every dev CP task. Watched with `jaah-forge` (task-authenticated) | verified: Jaah Deploy is the project's standard path (AC15). No console "restart CP" action exists, and no AWS write is used. | | AC13, AC14 (gate) | Worktree only (`./dc check full`, `tools/check_context.py`) | verified: nothing beyond the worktree. | Out of scope: - Do NOT convert routes or DB access to async (C2/C3/C4 own that); do NOT touch caps (C5/C6). - Do NOT set `PYTHON_GIL=0` to mask a GIL-enabling extension: fall back to 3.14t per the rule and record the package. Epic-wide (PRD §6, verbatim): - **Do not** promote to `master` or change prod's live config as part of done (Q8). Prod tfvars may be edited only so that the R7 budget test covers them. Do not apply them to prod. - **Do not** remove or loosen anonymous/public abuse controls (AS2): the WAF, `SharedRateLimiter` public and trigger-intake limits, read-relay `MAX_SOCKETS_PER_IP` / `ANON_*`, site-chat per-IP limits, headscale edge nginx `limit_conn` / `limit_req` (`infra/jaah/headscale/nginx.conf`). - **Do not** remove resource-safety bounds (AS1): body, frame and response size limits, read timeouts, `pool_timeout`. - **Do not** remove per-ECS-task DB pool sizing as a concept, or exceed RDS `max_connections` (AS3). - **Do not** add a numeric load-test gate or a benchmark harness. None was asked for (Q4, Q7). - **Do not** change node-side caps that are not on the CP-call path: vnode conntrack limits (`apps/jaahd/vnodehelper/`), MicroVM egress `MAX_CONNECTIONS_PER_TAP`, reconcile probe parallelism, `pr_stats_capture` semaphore. - **Do not** change unrelated product limits: plan and org limits (`orgs.limit_*`), `max_accounts_per_node`, node session caps, worktrees-per-node. - **Do not** introduce a leader-elected singleton CP or pin `desired_count` to 1 (ADR-0142 anti-pattern still holds). - **Do not** add a process-local registry consulted by a request path (ADR-0142 anti-pattern still holds). - Do NOT change task CPU/memory, RDS class or autoscaling max (operator D2: sizing out of scope). - Do NOT reintroduce multi-worker / multiple processes per container (operator D3), and do NOT keep a uvicorn fallback (D8). Done means: - C1-a — Dev task logs, for both the CP API service and the console-relay service, show one Granian process and the `free-threaded OK … GIL off` line, after deploy and after an ECS replacement (replaces PRD AC1) — oracle: dev ECS container logs right after the deploy and after a task replacement (the next merge's deploy rolls every task); quote the lines in the report. - C1-b — `worker_guard.assert_single_api_worker` still refuses N>1 (parsing Granian `--workers` and `GRANIAN_WORKERS`) and asserts `sys._is_gil_enabled() is False` at boot — oracle: unit tests for both, plus a gate test that imports `app.main` and asserts the GIL is off. `PYTHON_GIL=0` appears nowhere (`git grep -n PYTHON_GIL` → no setter). - C1-c — The proxy-header test passes: two different `X-Forwarded-For` values from 127.0.0.1 get separate `check_rate_limit` buckets; an `X-Forwarded-For` from a non-trusted peer is ignored — oracle: the new test in the gate. - C1-d — The WS keepalive test passes: a half-open node-bus / console-relay socket is detected on the 20 s ping/20 s timeout schedule, and the WS handshake carries no `permessage-deflate` extension — oracle: the new tests in the gate. - C1-h — The S5 launcher flags hold: `--workers 1`, a 10 s graceful shutdown (`--workers-kill-timeout`, below stopTimeout 15), access log off, WebSockets on. The ECS command and the local compose command both exec `./entrypoint.sh` — oracle: a test or a gate check over `entrypoint.sh`, and the dev log showing a graceful drain on the next deploy. - C1-i — Asyncio slow-callback logging (>100 ms) is on in dev, so a sync call on the loop is visible (Risk §6 mitigation, owned here) — oracle: config cited file:line, and a dev log line produced by a deliberate slow callback in a test. - C1-e — Free-threading race audit recorded: every R2 process-local item and lifespan actor listed in scope is marked safe or fixed in the PR body, each with file:line — oracle: PR body table; fixes carry tests. - C1-f — The quick and full gates are green on the new interpreter (the full suite runs on free-threaded CPython) — oracle: `./dc check quick` locally and the merge lane's `./dc check full`. - AC3 — | AC3 | Live console updates arrive exactly once on every CP task (cockpit_events fan-out). | Open the dev console Tasks board in two browser tabs, which the two dev CP tasks will likely serve from different tasks (re-read under operator D3: one process per task). Change a task's state in tab A. Tab B updates within 2 s, exactly once, with no duplicate rows or toasts. Repeat 10 times. | - AC5 — | AC5 | Node-bus commands reach nodes whichever CP task serves the request. | On dev, from the console, run a node command 20 times in a row, such as a worktree refresh or a node probe from Machines. All 20 succeed, with no "node not connected" error for a node whose heartbeat is live. | - AC6 — | AC6 | DBOS durable flows survive: unique executor per CP task, recovery intact. | Gate tests for `app/durable`. On dev, start a durable human-in-the-loop flow (an AskUserQuestion round), restart or redeploy the CP mid-flight, and answer. The answer is delivered and no workflow is stuck in PENDING (Audit or task view). | (Risk table: "DBOS on free-threaded … AC6 on dev after C1".) - C1-g (planner tightening, from Risk §6 "boots fine, then crashes or leaks under WS churn") — after deploy, a WS soak on dev: AC5 ×20, AC3 ×10 and one console PTY session of ≥5 minutes, with no CP task restart, no WS error frames and no memory climb visible in ECS memory utilisation over the soak — oracle: report records the runs and the ECS MemoryUtilization reading before/after. - AC14 (C1 part) — `python3 tools/check_context.py` passes. ADR-0142 states one Granian process per task on free-threaded CPython. `uvicorn` no longer appears in C1's own files: `apps/api/entrypoint.sh`, `infra/local/Dockerfile.api`, `apps/api/app/worker_guard.py`, `modules/ecs/variables.tf`, `modules/ecs/main.tf`, `infra/local/README.md`, the compose file, the ADR register and the API dependency manifest. The repo-wide grep is C8's — oracle: `git grep -niE uvicorn --` over those paths returns nothing. - S4 regression check — after this child deploys, the §8 burst probe on dev is no worse than the baseline at N=24 (`/api/tasks` N=24 max 4.6 s) — oracle: run the burst probe, record wall/p50/max in the PR body and report. Evidence, not a numeric gate; a child that makes it worse is not done. - AC15 — Jaah Deploy to dev succeeds for this child's merge commit on `develop`, and dev `/api/health` answers 200 after it — oracle: GitHub Actions "Jaah Deploy" run for the merge commit is green; `curl https://dev.jaah.ai/api/health` → 200. (If the deploy is cancelled/superseded by another deploy, leave it and complete.)

CP API and relay run one Granian process on CPython 3.14t, GIL off (3.15t blocked by bcrypt). PRs #7759 and #7762 merged; Jaah Deploy green, dev health 200, AC3 10/10, AC5 20/20. Burst probe at baseline but noisy. PTY soak and AC6 mid-flight unverified on dev. Filed #12842. Memory: proposed 1 gotcha.