Self-heal one-uid nodes, fleet sweep

## Original requirement (verbatim) > "Implement NodeVault on Windows and Mac" Operator decisions that widen it (PRD §3, verbatim answers): D1 "Parity: owner-only file"; D2 "macOS arm64 + Windows x64"; D3 "Real nodes on dev"; D4 "Console should not explain why something can't. It must make it work so that everybody can"; D5 "Every node, all OSes"; D6 "One is enrolled already" (superseded for the Mac by Architect OD3: no Mac is on dev — Mac live checks deferred). Architect decisions (settled, do not reopen): OD1 Linux self-heal on update; OD2 Windows widened ACL → repair and log (foreign owner / reparse point / hard link → refuse); AD1 no new uid-boundary mechanism; AD2 old-glibc/odd-arch Linux kinds are empty; AD3 macOS/Windows crypto bytes are `packaging/platform-pins.toml` bundle-time pins; AD4 `vendor_path.native_key()`; AD5/AD6 Windows explicit protected DACL via a `host_platform` seam; OD3 Mac live checks deferred. Full inputs (read them): PRD = Task 9748's prd artifact; Architecture = Task 9749's architecture artifact (§ numbers below refer to it). Epic: Task 9750's parent. ## This child: C3 — Self-heal one-uid misconfiguration on update, plus the AC-6 fleet sweep Requirements served (PRD §5, verbatim): - **R3 — Every supported node is capable (A1, A2).** Every node in A2's set reaches `capable = true` without operator action beyond the normal install or update. This includes one-uid Linux nodes (given a boundary per A1) and Linux hosts where today's vendored wheel won't load (old glibc or another machine the installer accepts). - **R5 — Upgrade path.** A node that registered `capable = false` under an older daemon becomes capable after a Jaah update, with no operator reset. The CP then delivers the org's live keys automatically (`ensure_grants`). Settled reshaping (Architecture AD1, AD2, OD1): every shipped installer already creates the uid split; old-glibc/odd-arch kinds are empty. The only one-uid nodes are misconfigured packaged nodes — repair them on update (operator: "Self-heal on update"). The daemon's capability rule stays as it is. ### Approach (Architecture §2.4) - **Runner user deleted (Linux):** already repaired by `postinst configure` (`packaging/postinst:99-102`). Add a test asserting it; nothing else. - **`JAAH_SESSION_USER` set empty on a packaged node:** `postinst configure` removes a `JAAH_SESSION_USER=` line whose value is EMPTY from `/etc/skynet/skynetd.env` and logs it. For a drop-in under `/etc/systemd/system/jaahd.service.d/` that sets it empty, postinst only logs a warning naming the file (never edits it — R-L1). Packaged nodes are never `sandbox` posture, so the sandbox exemption holds by construction. - Update the `environment.session_user()` docstring (`apps/jaahd/environment.py:435-439`): the empty-value opt-out is honoured only off the packaged install (dev, CI, sandbox); an update removes it on a packaged node. - **macOS / Windows parity:** the updaters (`packaging/macos/bin/jaah-update`, `packaging/windows/bin/jaah-update.ps1`) re-assert the installer's `JAAH_SESSION_USER` twin lines (`apps/api/app/node_bootstrap.py:1895`, `:2847`) when the value is empty — single-homed: reuse the installer's helper or `jaah-set-env`, never a copied literal. - After repair the daemon restarts, re-probes `node_capable()` and posts `capable=true` (Architecture §2.1 R5 path; no CP change). - Then run the AC-6 fleet sweep on dev (needs C1 and C2 merged and deployed; run the Windows live re-assert only after C2's AC-7 step is done, with auto-update off for win-t8703 during it and back on afterwards). Credentials & reach (Architecture §6 rows, verbatim): - UI checks for AC-2, AC-3, AC-5…AC-10 (Settings → NodeVault, Nodes, Credentials, Audit) and the AC-7 auto-update toggle — `org/console/dev-login`: tenant Owner/Admin on the dev CP UI `https://app.dev.jaah.ai` — **verified**: on roster, `credential_get` succeeded, `https://app.dev.jaah.ai/` answered HTTP 200. Its stored base URL host does not resolve, so use app.dev.jaah.ai - AC-9 org transfer, AC-10 plan term off, AC-6 cross-org node list (operator plane) — `me/jaah/debug/main/operator-login`: operator on dev — **verified**: signed in to app.dev.jaah.ai in a browser. Nodes (all orgs) listed 13 online nodes - The Windows node for AC-2/3/5/7/8/9/10 and C3(b) — `win-t8703` on dev: os windows, x64, online, 1.2.4150 (pre-change) — **verified**: present and online in the operator Nodes list. Used instead of a fresh cloud launch (PRD named "a Windows cloud node launched from the console"; an existing one serves and keeps AC-7's pre-change state). A fresh launch is therefore not needed - Runner-account shell on win-t8703 (`jaah nodevault …`, `type` as runner) — a Jaah console session on that node via the dev UI (runs as `.\jaah`) — **verified** as reachable UI (same credential/probe). The session itself is opened by C2 - **Elevated** step on win-t8703 (AC-5: add a Users/runner grant, then `icacls`) — an Administrator shell on the node — **deferred**: no admin shell reach was verified from this node. The console session runs as the runner. C2 proves the repair rule with unit tests over a fake DACL source. On the node it first tries the console's admin path; if none exists, it ASKS the operator for one round and otherwise records the live widen-and-repair step as deferred with that answer - C3(b) live re-assert on win-t8703 (setting the env value needs admin) — the same Administrator shell as AC-5's elevated step — **deferred**, for the same reason as that row. If no admin path exists, C3 records the live step as deferred with the operator's answer, and its oracle falls back to a unit test of the updater's re-assert logic - The Mac for AC-1, AC-4 and the Mac halves — a macOS node on dev — **deferred**: the operator answered "Defer Mac live checks". No macOS node is on dev (OD3) - AC-6 sweep: Ubuntu nodes and vnodes — the dev nodes listed above (multi-node hosts gmind-m8i-1-host, h3-host, pool-c-host, gmind-microvm-1-host and their vnodes) — **verified**: present and online in the operator Nodes list - AC-6 sandbox node — the local `./dc` stack in C3's own leased worktree — **verified**: worktree-only, no credential - C3(a) postinst repair — a Debian container on the builder (`docker run` in the worktree's toolchain) — **verified**: worktree-only. Root on a dev Linux node is **not** needed. Consequently the live "update a misconfigured dev node" step is **deferred**: no root reach on a dev Linux node was verified from here, and the container test is the oracle - Unit/gate oracles (AC-11, AC-12) — worktree only (`./dc test`, `./dc check full`, `python3 tools/check_context.py`) — **verified**: nothing beyond the worktree - AC-13 deploy — the PR's Jaah Deploy in GitHub, via `jaah-forge` (pre-authenticated) — **verified**: no extra credential Out of scope: Do NOT touch any of these (PRD §7, verbatim): - **N1** No OS secret store: no macOS Keychain, Windows DPAPI, CNG key storage or TPM binding (D1). Deferred follow-up: OS-keystore hardening for macOS/Windows. - **N2** No Intel (x86_64) macOS bundle and no Windows ARM bundle (D2). They are not supported nodes today. - **N3** No change to the CP's crypto: the envelope format, `nv:` key ids, RSA-4096/OAEP/AES-GCM constructions, `apps/api/app/credential_crypto.py`, `credential_keys.py`, ADR-0137 KMS tier, and the browser's key generation/backup in `apps/web/src/lib/nodeVaultKeys.ts` stay unchanged. Vectors in `apps/jaahd/tests/fixtures/nodevault_vectors.json` must keep passing as-is. - **N4** No new console copy or UI that explains why a node cannot hold keys (D4). The existing out-of-date-daemon label stays (A3). - **N5** No weakening of the passive-breach guarantee or the runner-never-gets-a-key claim to make a node capable (A1). - **N6** No change to which credentials NodeVault covers (node-consumed, non-OAuth2) or to CP-consumed credentials. - **N7** No EC2 mac cloud launch (D6). MicroVM guest sessions still see no keystore. The Multi-Node Host process stays excluded. - **N8** No change to the NodeVault plan feature or its per-org override semantics (A5). Architecture §7 additionally deferred: OS-keystore hardening (Keychain/DPAPI/TPM); rewriting operator systemd drop-ins (warn only). No CP (`apps/api`) or web (`apps/web`) change in this epic: the "Can't hold keys — Update Jaah" label (`apps/web/src/pages/settings/nodeVault/api.ts:169`) stays as the old-daemon fallback. Child boundary: no new uid-boundary mechanism (AD1); do NOT touch the NodeVault Python modules (`nodevault_*.py`, `cred_broker.py`, `secret_modes.py`, `vendor_path.py`) — only the `environment.session_user` docstring; do NOT edit operator systemd drop-ins; do NOT touch the compliance doc's keystore rows (C1/C2). Done means: - **AC-6 (R3, A1) No supported node is left out (minus the Mac).** Verify: on dev, compare Nodes (every online node running the current Jaah version) with Settings → NodeVault's node list. Observe: every such node, including the Mac, the Windows node, the packaged Linux node, each Linux edge-kind node, vnodes, and the local sandbox node on a dev stack, shows **Delivered**. None shows "Can't hold keys". On each formerly one-uid Linux node, a runner session cannot read the keystore (AC-4's read test, as the runner uid). Excluded: the Multi-Node Host process itself (its vnodes are counted), offline nodes (they show the waiting state and reach Delivered once back online), and nodes on an old daemon (AC-7). — oracle (scoped by the Planner): a table in the report listing every current-version online dev node (Ubuntu nodes, vnodes of gmind-m8i-1-host / h3-host / pool-c-host / gmind-microvm-1-host, win-t8703), its org, and its status. For a node in an org that has NodeVault set up (the `org/console/dev-login` org — name it in the report), the status must be **Delivered** in that org's Settings → NodeVault. For a node in an org WITHOUT NodeVault set up (never set up NodeVault in an org you do not own — it needs the owner's browser key generation), the oracle is `capable = true` with a registered identity fingerprint, read from the operator plane's node detail (`me/jaah/debug/main/operator-login`); if that page shows only the identity fingerprint, a registered fingerprint counts (the CP records an identity key only from a capable post — verify against `apps/api/app/routes/agent_nodevault.py` and cite it). Any node neither Delivered (set-up org) nor capable (other orgs) is a FAIL unless it falls under a named exclusion. Sandbox node: on this child's own local `./dc` stack (a local control plane, not dev) set up NodeVault in the local org and observe the sandbox node **Delivered** in the local Settings → NodeVault. The runner-uid read test on "each formerly one-uid Linux node" (deferred — no one-uid node exists on dev; proven instead by repair oracle (a)'s runner-uid read denial). The Mac (deferred — no macOS node on dev, OD3). - **Repair oracle (a) Linux:** a postinst test in a Debian container empties `JAAH_SESSION_USER` in `skynetd.env`, runs `dpkg -i` of the built `.deb`, and asserts the line is gone and the warning logged. Tightened by the Planner: the same test asserts a NON-empty `JAAH_SESSION_USER` value and every other line of `skynetd.env` survive byte-for-byte, and that an empty-value drop-in is left unedited with a warning naming it. It also asserts the repaired node has a uid boundary: under the repaired env `environment.session_user()` resolves to `skynet` (`session_user.switch_needed()` True), and a read of a `0600` file in the daemon's keystore dir as the `skynet` uid is denied. The live "update a misconfigured dev node" step (deferred — no root reach on a dev Linux node verified; the container test is the oracle). - **Repair oracle (b) Windows, live on win-t8703:** empty `JAAH_SESSION_USER` with `jaah-set-env` from the console session, run the console's Update action, and observe the value restored and the node Delivered. If it cannot be run (no Administrator path — Architecture §6), record it as deferred with the operator's answer, never as passed; then a unit test of the updater's re-assert logic is the oracle. - **Repair oracle (c) macOS:** the runnable `jaah-update` harness in `tools/tests/test_macos_node_scripts.py` (the tests from line 231) gains a case: an empty value is re-asserted (and a non-empty value is left unchanged). - **Runner-user test:** a test asserts `postinst configure` re-creates `skynet` on upgrade. - **AC-12 Linux behaviour does not regress.** Verify: the jaahd NodeVault suites (`apps/jaahd/tests/test_nodevault_*.py`, `test_vendor_native.py`) and `./dc check full` pass in the merge lane. Observe: green. - **AC-13 Deploy.** Observe: the PR(s)' Jaah Deploy to dev succeeds, and the dev node bundles for macOS and Windows carry the change (`jaah --version` on each node shows the deployed version) before AC-1–AC-10 are run. — Windows half; macOS (deferred — OD3).