Remove runner and node caps

Task 12735 · Done · Coder · created · updated

## Original requirement (verbatim) > "Make the control-plane API highly concurrent and distributed: many processes, no per-task connection caps." Epic 12693. Settled upstream: PRD (task 12694) and Architecture (task 12695). The operator overrode the PRD's multi-worker reading (D3): ONE Granian process per task on free-threaded CPython, async DB I/O on one shared pool. The full design (decisions D1–D9, risks, oracle tooling) is the `architecture` artifact of task 12695; read it (`artifact_read` / `task_read` on 12695) for the pinned seams S1–S5 (§4), the measured cause (§1), material risks (§6) and the burst-probe tooling and baseline (§8) before you start. This body carries this child's scope, acceptance and fences verbatim. ## Scope of this child: C5 — Runner/node caps (R3) (Architecture §5, verbatim) ### C5 — Runner/node caps (R3, unchanged from PRD) - `apps/jaahd/cred_broker.py`: - Remove the per-peer in-flight refusal (`_MAX_INFLIGHT_PER_PEER`, :254) and the node-wide 8 refusal (:253). - The blocking-call executor becomes large (64 threads) or the call path becomes async httpx. Pick async if the broker already has a loop, which it does for the wait lane. - Remove the wait-lane per-peer 3 and the node cap (`_MAX_WAIT_PER_PEER`, `_wait_node_cap`, :297-319). - The accept bound `_MAX_CONNECTIONS` (:257) becomes an AS1 fd bound of 1024, with a comment naming the resource. - `apps/api/app/node_bus.py:66`: `MAX_PENDING_PER_NODE` becomes an AS1 memory bound of 4096 with that comment. - Remove the `node_max_concurrent_commands` setting from the catalog (`operator_settings/catalog_runners.py:503`) and from the jaahd node settings, and its semaphore (`apps/jaahd/commands.py:294-305`). Any stored value is ignored, and an additive migration deletes the row. - jaahd ships in `apps/jaahd/**`, which the CE mirror tracks: cite packaging paths as prose, not backticks (mem-32955). - **Acceptance:** PRD AC7. PRD R3 (verbatim): **R3 — No runner/fleet concurrency caps (A2).** No per-runner, per-fleet-task or per-node *concurrency-count* cap remains on the runner → node → CP path (jaahd broker in-flight, wait lane, per-node wait cap, CP per-node pending, `node_max_concurrent_commands`). Exception: an AS1 safety bound sized so that no legitimate workload ever reaches it. ## Approach (planner) The broker already has a loop for the wait lane, so make the call path async httpx there. Keep the AS1 bounds with a comment naming the resource. ## Applies to every child (verbatim) Each child is one implementer's PR. **Dependencies are data/API only.** Children that edit the same file are not serialized; they rebase at merge. **Every child owns AC15:** its merge's Jaah Deploy must go green, and dev `/api/health` must answer 200. Every child that changes request-path behaviour also runs the S4 regression probe. ## Constraints (PRD §7, verbatim) ## 7. Constraints - Merges go through the merge lane on `./dc check full`. Each merge's Jaah Deploy must be watched to green (project instruction). - Migrations, if any, must be additive while two releases overlap (`docs/standards/coding-standards.md`). - The deploy overlap doubles tasks, so every connection-budget formula carries ×2. - Live state is read through the console UI only (CLAUDE.md hard rule). `tools/cp_psql.sh` was refused by the credential broker for this BA run (`org/aws/cp-psql-reader`), so current dev latency was not measured here. The Architect must get it (AS6, AC12). Credentials & reach: | Oracle | Credential / identity / environment | Status | |---|---|---| | Burst probe, AC3, AC5, AC8, AC10 (console as operator) | `me/jaah/debug/main/operator-login` on `app.dev.jaah.ai` | verified: `credential_get` via broker OK. Login 200 and timed GETs from this node (ts node of task 12695). | | AC4, AC7 (Task action posting a PR comment; 3 runners on one node) | Same operator login, "Act as Gmind": project `mr-6382-static` (repo jaah-6382-backend), node ts2-node6 | verified: as the operator on dev, `GET /api/projects` lists `mr-6382-static` and `GET /api/nodes` lists `ts2-node6` (200, from this node). Launching runners is a console action on that project. **The AC4 comment check runs on a throwaway draft PR** that the builder opens on that repo for the test and closes afterwards, never on client work. | Out of scope: - Do NOT change node-side caps off the CP-call path: vnode conntrack limits, MicroVM egress `MAX_CONNECTIONS_PER_TAP`, reconcile probe parallelism, `pr_stats_capture` semaphore (PRD §6). - Do NOT change `max_accounts_per_node`, node session caps or worktrees-per-node. - Do NOT remove resource-safety bounds (AS1); keep body/frame/response size limits and timeouts. Epic-wide (PRD §6, verbatim): - **Do not** promote to `master` or change prod's live config as part of done (Q8). Prod tfvars may be edited only so that the R7 budget test covers them. Do not apply them to prod. - **Do not** remove or loosen anonymous/public abuse controls (AS2): the WAF, `SharedRateLimiter` public and trigger-intake limits, read-relay `MAX_SOCKETS_PER_IP` / `ANON_*`, site-chat per-IP limits, headscale edge nginx `limit_conn` / `limit_req` (`infra/jaah/headscale/nginx.conf`). - **Do not** remove resource-safety bounds (AS1): body, frame and response size limits, read timeouts, `pool_timeout`. - **Do not** remove per-ECS-task DB pool sizing as a concept, or exceed RDS `max_connections` (AS3). - **Do not** add a numeric load-test gate or a benchmark harness. None was asked for (Q4, Q7). - **Do not** change node-side caps that are not on the CP-call path: vnode conntrack limits (`apps/jaahd/vnodehelper/`), MicroVM egress `MAX_CONNECTIONS_PER_TAP`, reconcile probe parallelism, `pr_stats_capture` semaphore. - **Do not** change unrelated product limits: plan and org limits (`orgs.limit_*`), `max_accounts_per_node`, node session caps, worktrees-per-node. - **Do not** introduce a leader-elected singleton CP or pin `desired_count` to 1 (ADR-0142 anti-pattern still holds). - **Do not** add a process-local registry consulted by a request path (ADR-0142 anti-pattern still holds). - Do NOT change task CPU/memory, RDS class or autoscaling max (operator D2: sizing out of scope). - Do NOT reintroduce multi-worker / multiple processes per container (operator D3), and do NOT keep a uvicorn fallback (D8). Done means: - AC7 — | AC7 | No runner/fleet concurrency-count cap remains (R3). | `git grep -nE "_MAX_INFLIGHT_PER_PEER\|_MAX_WAIT_PER_PEER\|_wait_node_cap\|MAX_PENDING_PER_NODE\|node_max_concurrent_commands"` returns no live enforcement, or only an AS1 safety bound with a comment naming its resource reason and a value at least 10× observed peak. On dev, run 3 runners on one node, each issuing ≥5 concurrent control tool calls (e.g. `task_get` in parallel subagents). No call returns `busy` or a cap refusal. | - C5-a — `node_max_concurrent_commands` is gone from the operator catalog and the jaahd node settings; an additive migration deletes any stored row; a stored value is ignored — oracle: tests + `git grep -n node_max_concurrent_commands` shows only the migration. - C5-c — The remaining AS1 bounds are pinned: the jaahd accept bound `_MAX_CONNECTIONS` = 1024 and `node_bus.MAX_PENDING_PER_NODE` = 4096, each with a comment naming the resource it protects (fds, memory). A test asserts each value — oracle: tests plus `git grep -nE "_MAX_CONNECTIONS|MAX_PENDING_PER_NODE"`. - C5-b — The quick gate (including mirror-tracking-refs) passes with apps/jaahd changes — oracle: `./dc check quick`. - AC15 — Jaah Deploy to dev succeeds for this child's merge commit on `develop`, and dev `/api/health` answers 200 after it — oracle: GitHub Actions "Jaah Deploy" run for the merge commit is green; `curl https://dev.jaah.ai/api/health` → 200. (If the deploy is cancelled/superseded by another deploy, leave it and complete.)

Outcome

Removed broker in-flight/wait caps and node_max_concurrent_commands (+migration); AS1 bounds pinned (fds 1024, pending 4096). PR #7752 merged fa8bb743; Jaah Deploy green, dev health 200. Live 3-runner AC7 run not done (needs new jaahd on node).

Pull request

Artifacts

Activity

  • Draft
  • Ready
  • Runner started
  • Runner ended (done)