Console network choke point core

## Context (BA Task 12447 → Architecture Task 12448 → Plan Task 12449) Original request (verbatim): "Regarding task <https://app.jaah.ai/tasks/12393> This is a repeat situation with the same user apparently behind a broken corporate firewall. He says that previous fix helped and his task list is no longer causing trouble, but he was now struggling with Projects edit. Go over all UI surfaces and ensure that the same code is reused everywhere - one choke point - that works identically and uses the approach and protocols that are now proven working. If there will be other similar issues we would only have to fix it in one place" Root cause (Architecture §0, AC-0): the hang was a WRITE — `PUT /api/projects/54` sat on a dead direct HTTP/2 connection (prod `activity_events` 1661992/3; every page read was relay-rescued, `client_events` 1335-1345). Writes are bare `fetch` with no relay/timeout (`apps/web/src/api.ts:68`); the relay is GET-only (`apps/api/app/routes/read_relay.py:13`). Operator decisions (verbatim, PRD §3 + Architecture §1): D2 "**Relay once + idempotency key.** Every write carries a unique key. When the direct connection looks dead, the write also goes over the WebSocket relay, and the server applies a given key at most once." D3 "**All four:** all `api()` reads + writes; the raw-fetch stragglers (`useVersionCheck`, live `config.ts`, live rewriter, S3 upload PUT); streams + terminal (EventSource `/api/stream`, terminal WebSocket); browser-fetched URLs (`apiHref`: img, iframe, downloads)." D4 "**Yes, also unify edit pages:** load via `useEditRecord`, save via `useSubmit`". A1 one epic, parallel where no dependency. A2 **At most once** (claim before run). A3 **All console writes, capped** (body ≤ 1 MiB, per-session write budget, key required). A4 **Public relay, live paths only** (cookieless socket, `GET /api/public/live/*`, per-IP budget). Full documents (read your section in full before building): the PRD and Architecture are the settled `prd` / `architecture` artifacts of Tasks 12447 / 12448 (read with `task_read` / `artifact_read`). Your section below is copied from them; the pinned seams are binding contracts other children build against — do not change them without updating the sibling that consumes them. ## Your scope: C3 — Choke point core (`apps/web/src/net/`) Create `apps/web/src/net/` (send, relay [git mv of `lib/readRelay.ts`], stall [git mv of `lib/requestStall.ts`], errors) with the pinned `send`/`SendInit` exports; put `api.ts` on top; write path per §3.3 rules 1-7 against the C1 idempotency contract and the C2 relay v2 frames; move the three stray `.json()` sites (`components/ConsolePane.tsx:122`, `lib/supportChat.ts:113`, `components/FullDiskAccessStep.tsx:21`) onto `api()`/`readJson` (Architecture §3.6, last bullet). Then run every dev oracle you own. Capture the AC-3 healthy-save baseline on dev BEFORE opening the PR (Architecture §4 baselines, verbatim): **Baselines before merge.** AC-3 needs the median save time and AC-9 needs side-by-side screenshots, both against develop. Once a child's merge deploys, dev runs the new code. So each of these children captures its baseline on dev **before** opening its PR and attaches it to its report: - C3: 20 Project Saves on a healthy connection, timed in Playwright. - C6: desktop and phone-width screenshots of each AS7 page. Every coder child runs the dev oracles of the ACs it owns against `https://app.dev.jaah.ai` after its merge deploys (see §6), not `dev.jaah.ai`. That PRD host does not resolve from node h2. ### Architecture §3.3 (verbatim) ### 3.3 The choke point (web) — `apps/web/src/net/` [child C3, extended by C4/C5] **Module layout (pinned exports, the single home):** - `net/send.ts`: `send(path, init): Promise<Response>`. It holds the logic that is today `apiFetch` (`api.ts:65-73`). It applies the session headers, then the rewriter if one is installed, then `raceRequest`. `api.ts` keeps its JSON helpers (`api`, `get`, `post`, `put`, `patch`, `del`, `postForm`, `getPage`, `getText`, `readJson`, `refuse`, `apiDetail`) and calls `send`. `apiFetch` stays as a re-export so callers do not churn. - `net/relay.ts`: today's `lib/readRelay.ts`, moved. `raceRead` becomes `raceRequest(path, init, headers, direct)`. GET semantics are **byte-for-byte unchanged** (PRD out-of-scope). The write path is new (below). `relayState()` and `resetReadRelay()` keep their names, re-exported from the old path until the import sites move in the same PR. - `net/stall.ts`: today's `lib/requestStall.ts`, moved. `watchWait` times **every** method. A non-GET reports `slow_write` (≥ 2 s) or `stall`, with `detail.method`, `path`, `outcome` (`direct|relayed|replayed|failed`), `latency_ms` and `relay` (AC-11). - `net/errors.ts`: `export class NoAnswerError extends ApiError` (status 0). It carries the copy `NO_ANSWER = 'no answer from the server. Nothing was lost on this page — retry'`. That wording is the C3 implementer's to tighten under `docs/standards/ui-copy-standards.md`, but there is one constant, read through `apiDetail`. So the visible message is the page's own fallback joined to it: "Failed to save project — no answer …". **`send` signature (pinned; C4 and C5 build against it):** ```ts export type SendInit = RequestInit & { /** 'session' (default): the console's relay socket. 'anon': the cookieless live socket. 'none': never relayed. */ relay?: 'session' | 'anon' | 'none' /** Overall bound in ms. Default: writes 30_000; reads unbounded, as today. */ boundMs?: number /** true: no session headers, no rewriter, no refusal side effects (a static file such as /version.json). */ raw?: boolean } export function send(path: string, init?: SendInit): Promise<Response> ``` **Idempotency answers on the client (§3.1 → what the user sees):** - `409 idempotency_in_progress`: not an answer. Keep waiting on the other copy, inside the 30 s bound. - `409 idempotency_outcome_unknown` and `409 idempotency_replay_unavailable`: final. They surface through `apiDetail` with the copy constant `OUTCOME_UNKNOWN` = "may have saved — reload to check", in `net/errors.ts`. The form keeps its draft. - `422 idempotency_key_reused`: a client bug (one key is minted per call), so it can only happen if this module is wrong. It is final, surfaces as an ordinary refusal, and is reported as `uncaught_error` telemetry. - A replayed answer (`Idempotent-Replayed: true`) is used exactly like a direct one. **Write path rules (AC-1, AC-3, AS1, AS3):** 1. `send` adds `Idempotency-Key` to every non-GET `/api` request, minted once per call. It is never re-minted for the relay copy. 2. Healthy connection (`raceAtOnce` false and the direct answers): sent once, direct, no timer cost. The relay is not touched. 3. `raceAtOnce` true (suspect, or quiet on a dropping network: Nick's case on every save) → the relay copy is sent **at once** beside the direct copy. Otherwise, after `RACE_AFTER_MS` with no direct answer, the shared probe runs (`probeDirect`). If the probe goes unanswered, the relay copy is sent. If the probe is answered, the request is just slow and waits on direct, re-checked every `RACE_AFTER_MS`. These are the same rules as reads. 4. The first usable answer wins. The relay copy runs on up to `MAX_RELAY_ATTEMPTS` sockets with the same key, which is safe under C1. A relayed `401`/`403` never wins over the direct copy, as for reads. A relayed `409 idempotency_in_progress` is not an answer: keep waiting on whichever copy is still open. 5. **Overall bound `WRITE_BOUND_MS = 30_000`.** If neither copy has answered, the direct copy is aborted and `NoAnswerError` is thrown. `useSubmit` already routes it to `formError`. The form's draft is untouched and the leave guard still applies, because the save never resolved. 6. `FormData` bodies are serialized for the relay with `new Response(form)` (bytes plus the boundary content-type). Bodies over 1 MiB are not relayed. They still get the 30 s bound and the key. 7. The direct `fetch` and the relayed answer both pass through `readJson`, so the unreadable-2xx rule is identical for both. ### PRD assumptions that bind this child (verbatim) - AS1. **Write wait bound.** Assumed a write that neither the direct connection nor the relay answers fails visibly within **30 s** with a "Not saved — retry" message, keeping the user's draft. Why: the operator's complaint is the 5-minute hang; D2 makes a relay answer the normal case, so 30 s is only the both-dead case. The architect may tighten it. - AS2. **Idempotency covers every console write.** Assumed every `POST/PUT/PATCH/DELETE` the console sends to `/api` carries the key and is deduplicated by the server, not only Projects. Why: D3 "all api() reads+writes" and A5 "fix it in one place". - AS3. **The race trigger for writes reuses the read relay's liveness rules** (dead-connection proof, suspect/dropping memory). A write on a healthy connection is sent once, direct, with no extra latency. Why: A4 "protocols that are now proven working". - AS4. **Intentional internals stay where they are, but are named.** The relay's own probe and socket, client telemetry and the live beacon (`keepalive` fire-and-forget) may keep their own `fetch`, as entries on the gate allowlist (AC-4), each with a one-line reason. Why: they are the choke point's own machinery, or must not be raced. - AS8. **No feature flag.** The change is on by default in dev and prod, like #7651. ### Requirement constraints (PRD §7, verbatim) - Writes are never applied twice (D2, AC-2). Idempotency touches every mutating route, so it is a security-surface change (data integrity). Project instructions grant merge permission. The review lane still applies. - Act-as headers (`lib/actAs.ts`) and session headers (`lib/activeOrg.ts`) must ride relayed writes exactly as direct ones. A relayed write must be authorized as the same session. A relayed 401/403 never wins over the direct answer, the same rule as reads. - Single-home: the relay rules live in one module. `docs/standards/coding-standards.md` and the context policy apply. ### Risks that bind this child (Architecture §5, verbatim) 3. **Healthy-path latency (AC-3 ±10 %).** The trap is relaying every write "to be safe". The rules in §3.3 send the relay copy only under the read relay's dead-connection proof. 4. **Relayed answer vs. the leave guard.** A `useSubmit` success from a relayed copy must navigate exactly like a direct one. `send` returns a `Response` either way, so no caller can tell. C3's vitest pins this. 5. **Key reuse across user retries.** Re-using a key on "retry" after a 5xx would replay the 5xx forever. Keys are minted per `send` call, never stored on the form. 8. **`readRelay.ts` move.** The read path must stay behaviourally identical (PRD out-of-scope). C3 moves the file with `git mv` and keeps the existing `readRelay.test.ts` passing unchanged, apart from imports. ### Acceptance criteria (PRD §5, verbatim) **AC-0 — Root cause confirmed before build.** The architecture stage reads Nick's prod `client_events` for the Projects-edit hang and names the request(s) that hung: method, path, duration, relay state. - verify: the architecture spec cites the event rows (time, kind, path, `relay.dropping`). - States: the hang was a write → AC-1 covers it. The hang was something else (a read, the chunk, a stream) → the spec adds that path to the choke point and to AC-1's oracle. No events recorded → the spec says so, and AC-1..AC-3 still hold. **AC-1 — Projects edit Save never hangs on a dead connection.** With the direct connection dead for writes, changing a field on `/projects/<id>` (General tab) and clicking Save shows the normal success state and navigates as today, within **10 s**. Reloading the page shows the new value. - verify: dev Playwright: open `https://dev.jaah.ai/projects/<id>`, hold every direct non-GET request to `/api/projects/*`, edit Description, click Save. Observe the success state and navigation in ≤ 10 s. Reload and observe the value persisted. - Same oracle for each tab write: Slots tick, Environment add/edit/delete a variable, Memory retire, Merge lane Stop, Benchmark Save, Production QA Save, Live Cast save/publish, Budget Save. Each completes in ≤ 10 s under the dead connection. **AC-2 — A write is applied exactly once.** A write sent both direct and over the relay (or retried) with one idempotency key changes server state once and returns the same answer to both copies. - verify (API test): send `PUT /api/projects/{id}` twice with the same key → one `updated_at` change, one audit row, identical bodies. Send a create (`POST`, e.g. env var) twice with one key → exactly one row. Send two writes with different keys → both applied. Reuse a key with a different body → refused (4xx), nothing applied. - verify (dev): AC-1's flow while the direct request is released late (it lands after the relay). Observe one audit entry for the save. - States: direct only; relay only; both arrive (either order); key reused with different payload; key expired (the retention window the architect sets). **AC-3 — Every write ends in a bounded, visible outcome.** States and required outcome: - healthy connection → sent once direct, no added latency (median save time on dev unchanged ± 10 % vs develop); - direct dead, relay alive → success ≤ 10 s (AC-1); - direct and relay both dead → "Not saved — retry" (wording per `docs/standards/ui-copy-standards.md`) within 30 s (AS1). The draft stays in the form; the leave guard still warns; - server refusal (4xx) → the same field or page error as today; - server error (5xx) → the same error as today; it is not silently retried into a duplicate. - verify: dev Playwright for each state (hold direct; hold direct + block `/api/read-relay`; trigger a validation refusal). Observe the outcome and the time. **AC-8 — Same behaviour on every surface.** The rules (unreadable 2xx body throws; reads raced; writes relayed with a key; bounded failure) apply identically wherever a request starts: an edit page, a list, a dialog, a tab, the live host. - verify: vitest on the choke-point module covers each rule once. No page-level copy of these rules remains (grep: no `resp.json()` outside the choke point except the allowlist). **AC-10 — On by default, deployed.** No flag or setting gates any of this. After merge, the Jaah Deploy to dev succeeds and AC-1 passes on `dev.jaah.ai`. - verify: the GitHub deploy run for the merge is green; AC-1's flow on dev. **AC-11 — Visible in telemetry.** A relayed or failed write records its outcome in the existing client report (`relayState` → `clientTelemetry.ts`) for flagged orgs/users, so the next incident shows whether a write was rescued. - verify: dev with telemetry on for the test user, run AC-1 → a `client_events` row shows the relayed write (method, path, outcome, latency). Credentials & reach: (Architecture §6, verbatim rows) | Oracle | Credential / identity / environment | Status | |---|---|---| | AC-0 telemetry (done here) | `me/jaah/system/cp-psql-reader` via `CP_AWS_CREDENTIAL` → `tools/cp_psql.sh main` (read-only `cp_readonly`) | **verified**: `\d client_events` and the event reads above returned | | AC-11 rows on dev | same credential → `tools/cp_psql.sh dev` | **verified**: `select count(*) from client_events` on dev returned | | AC-1/3/5/6/7/9 dev Playwright | `me/jaah/debug/main/operator-login` (operator; its description says it works on dev and main) on `https://app.dev.jaah.ai` | **verified**: `credential_get` succeeded; `GET /version.json` → 200 from h2. `org/console/dev-login` also exists, but its base_url names a host that does not answer from h2, so the operator login is the one to use. | | AC-11 turning telemetry on for the test user | the operator login above, **through the console UI**: the org switch on `/customers/{org}?tab=overrides`, or the member switch on `/users/{id}?tab=system` (linked from Audit → Client errors, `ClientErrorsTab.tsx:78-79`). Never a raw API call (CLAUDE.md "Operate Jaah only through the Jaah UI"). | **verified** (credential); the page exists in code | | AC-10 deploy run | `jaah-forge pr checks` / `ci log` (pre-authenticated forge) | **verified**: no credential to fetch | Out of scope: PRD §6 (verbatim) — - Do NOT add a hostname, port, certificate, or change cookies or auth to carry writes. The WebSocket relay carrier stays (decision of Task 11936). - Do NOT change the read relay's existing read semantics (race timing, probe, suspect and dropping memory) beyond what writes, streams and uploads need. - Do NOT change any edit page's visible UI: layout, fields, tabs, copy, URLs, routes (AC-9 quality). - Do NOT change React Query's global defaults (`apps/web/src/queryClient.ts`, `retry: false` for queries). - Do NOT change server route business logic. Server changes are limited to (a) idempotency for console writes and (b) the relay accepting writes under that key. - Do NOT touch `apps/site-widget`, `apps/docs`, `apps/jaahd`, `packages/` (AS9). - Do NOT refactor non-edit-page mutations (compliance, billing, playbook list, template cards) beyond what routing them through the choke point needs. - Do NOT add a feature flag (AS8). - Do NOT treat this as Nick-only: no per-user or per-org switch. - Do NOT touch streams/terminal/`apiHref` sites (C5), stragglers/live host/`putToStorage` (C4), the gate script (C7), or edit-page load/save structure (C6). - GET relay semantics byte-for-byte unchanged; `readRelay.test.ts` passes with import changes only. Done means: - AC-0 — oracle verbatim above (PRD §5 AC-0 `verify:` lines) - AC-1 — oracle verbatim above (PRD §5 AC-1 `verify:` lines) - AC-2 — oracle verbatim above (PRD §5 AC-2 `verify:` lines) - AC-3 — oracle verbatim above (PRD §5 AC-3 `verify:` lines) - AC-8 — oracle verbatim above (PRD §5 AC-8 `verify:` lines) - AC-10 — oracle verbatim above (PRD §5 AC-10 `verify:` lines) - AC-11 — oracle verbatim above (PRD §5 AC-11 `verify:` lines) - AC-0 cross-check: report cites the §0 rows and confirms the first slow_write/relayed-write row now appears for a held PUT on dev. - AC-2 here = its dev oracle (direct released late → exactly one audit entry); API-test part is C1's. - AC-1 covers the main Save AND every listed tab write, each ≤ 10 s with the direct connection held; a passing Save with any tab still hanging FAILS this child. - AC-3 both-dead state shows the readable 'not saved — retry' message within 30 s with the draft intact and leave guard active (screenshot in report); healthy median within ±10 % of the baseline. - AC-2 relay seam (server pytest against the real app, C1 middleware + C2 relay): a real `PUT /api/projects/{id}` and an env-var `POST`, one copy over `/api/read-relay` and one direct with the same key, in BOTH orders, plus relay-only → exactly one apply, identical bodies, `idempotent-replayed: true` and `location` present in the relay answer frame, and the relayed copy is judged as the same console session. - Dev records: create a scratch project through the console UI for these oracles (or use one you created); restore every setting you change afterwards. Never unpublish the single published dev Live Cast project (C4's oracle needs it) — use the scratch project for Live Cast save/publish. Never stop a merge lane another project relies on. - No bypass left in the files you own: grep the Architecture §3.6 patterns (`fetch(`, `new EventSource`, `new WebSocket`, `XMLHttpRequest`, `navigator.sendBeacon`, `apiHref(`, `.json()` on a Response) — zero hits outside `apps/web/src/net/`, `api.ts` and the §3.6 allowlist (clientTelemetry, live beacon, lazyRetry). Cite the grep output in the report. - `./dc check quick` green; PR merged via the merge lane; Jaah Deploy to dev green. Dev oracles run on `https://app.dev.jaah.ai` (the PRD's `dev.jaah.ai` does not resolve from the nodes — Architecture §7), after your merge's Jaah Deploy to dev succeeds. Watch that deploy; if it fails, fix and merge again.