NodeVault rotation, rekey and retire workflows

## Original requirement (epic 5269, verbatim) Build NodeVault feature: - Is a product that can be activated on a billing plan (once it lands in 5246) - When NodeVault is activated for an org, Org Admin can start Setup in Settings / NodeVault, that initiates a stepper: - Explanation with a diagram - On the backend System generates a Temporary NodeVault Key and sends it to all nodes (encoding each with node key) in the organization. Nodes store it in a secure place (next to other sensitive information, such as node token) - Generate NodeVault Key / Upload NodeVault Key. When Generate NodeVault Key - browser generates a key-pair (same algo as Temporary) encodes it with Temporary NodeVault Key public key and sends to CP to fan out to all nodes. Stores public key in the DB - Migrate credentials to NodeVault: all, select. Backend goes through all credentials that user selected and that are KMS encoded, decodes them and re-encodes them with NodeVault Key public key. No cred logging. - Show the result to the user - Credentials get a checkbox NodeVault (on by default for each new cred) - Credentials remember their current NodeVault key hash - When a broker requests a credentials and the credential is NodeVault, the broker uses the private NodeVault Key to decode it - Admin can rotate the NodeVault in Settings / NodeVault. The rotation procedure is a stepper: - Generate / Upload - same as at Setup - Migrate credentials: some of the creds may still be KMS - offer to migrate them - The new key is encrypted with current key (in browser) and fanned out to all nodes. Nodes decode and store the new key, keeping the old one - CP goes through each key and sends them to nodes to rekey. Nodes decode the cred, reencode it with new public and send them back to CP - All these workflows must be resilient and retry when something fails. Use DBOS to make it as reliable and idempotent as it gets Goal: reliable, provable way of Jaah not seeing and not being able to obtain access to client's credentials with all paths hermetic and auditable. ## Settled inputs (binding — realize, do not reopen) Full documents: BA PRD = Task 5273's `prd` artifact; Architecture = Task 5274's `architecture` artifact (`task_read` 5273 / 5274, then `task_read_artifact`). READ the architecture sections this child names — §2 crypto constructions, §3.2 data model, §4 interface contracts are PINNED and every child builds against them exactly. Binding decisions: D1 node-only (CP-consumed creds stay KMS) · D2 per-node identity key pairs (NIK) replace the CP temp key · D3 Jaah-hosted nodes included · D4 mandatory passphrase backup, no Jaah escrow · D5 later-joining nodes get the key automatically from a peer · D6 NodeVault is a plan parameter with per-org override · D7 OAuth2 excluded · D8 old keys kept until admin retires (refused while any credential uses it) · D9 passive-breach-proof + audited, no claim vs malicious Jaah · D11 plan off after setup ⇒ existing NV creds keep working, setup/rotation/migration/retire/new-NV frozen · D12 Test disabled for NV · D13 rogue-node enrollment is detectable (audit + node list), not impossible · E1 vendor `cryptography` into jaahd · E2 rotation chain-wraps new key under CURRENT key (K1: rotation does not recover from a leaked current key) · E3 DBOS for every workflow (carve-out from ADR-0074) · E4 AC-9.3 dropped (no admin notification on key delivery) · E5 node key files daemon-owned 0600 in a 0700 dir. Pinned crypto (arch §2): RSA-4096 e=65537 RSAES-OAEP SHA-256/MGF1-SHA-256 everywhere; AES-256-GCM 12-byte nonce; key hash = hex(sha256(SPKI DER)); envelope key id `nv:<hash>`; secret envelope = ADR-0137 v2 tier-2 unchanged; key blob `{v:1,k,alg,ek,n,ct}` AAD `nodevault.key|v1|<org_id>|<key_hash>|<recipient>`; backup PBKDF2-SHA256 600k. SECURITY SURFACE: this epic is credentials/crypto. Never log/checkpoint/return a secret value, private key, data key, passphrase or backup. The merge needs the security review lane or human approval per branching policy. Quality bar: "technically passes" is the floor. A build that satisfies the words but leaks key material to the CP, marks a step done before the node acknowledged it, or shows raw ids/stack traces to the user FAILS this child. ## This child — scope (HOW) Architecture child C5, ROTATION half (arch §3.4 rotate/rekey/retire, §3.5 abandon, §4.1). On top of the setup/delivery children's `durable/nodevault.py` and `routes/nodevault.py`: - `POST /api/nodevault/rotation {public_key, source, backup_confirmed:true, chain_blob}` → 202 {operation_id}; key_hash recomputed; audit key generated/uploaded + backup confirmed. - `nodevault-rotate:{op_id}`: distribute (new key `pending` + chain blob; ensure_grants for every capable node — holders of current resolve to `chain`, others `peer`) → promote (wait until every capable node ONLINE at check time has the new key delivered; ONE txn new→current, old→previous) → migrate_choice → migrating (targets the NEW key) → rekey → result. Abandon allowed only before promotion (§3.5). - Rekey phase: batches ≤50 credentials whose k names the previous key; `nodevault.rekey` on an online node holding both keys (wait HOLDER_ONLINE if none); swap ek and k per item in one txn ONLY if k still names the old key (idempotent); re-query until zero remain (catches seals that raced promotion); audit "credential rekeyed (old → new hash)". - `POST /api/nodevault/keys/{hash}/retire` + `nodevault-retire:{op_id}`: refused at API and in a step unless key is `previous` and no credential's k names it (409 "N credentials still use this key"); key → retiring (wire then refuses seals under it); grants → retire_pending; wait GRANT_RETIRED (24h re-check); key `retired` only when every grant retired; offline nodes keep the op retiring with a pending count; retire abandon restores `previous` + re-delivers (§3.5). Audit "old key retired". - Plan freeze applies (rotation/retire refused when frozen). ## Acceptance criteria carried by this child (PRD, verbatim unless marked amended) - AC-6.2 (A14). The new private key reaches nodes only in a form the CP can't open (chain-wrapped under the current key in the browser, E2). Nodes store the new key **and keep the old one**. *Verify:* no request carries the new private key in a CP-openable form; on a node both key hashes are listed by `jaah nodevault keys`. - AC-6.3 (A13). The Migrate step lists eligible credentials still on KMS and offers to migrate them straight onto the **new** key (same rules as AC-4.7). - AC-6.4 (A15). Rekey. The CP sends each NodeVault credential sealed under the old key to an online key-holding node. The node opens it, re-seals it under the new public key, and returns only ciphertext. The CP stores it, and the credential's key hash changes. This is idempotent per credential: a rerun skips credentials already on the new key, and a duplicate return is harmless. *Verify:* rotate with 5 NodeVault credentials, killing the CP process (or the node) mid-way. After recovery all 5 show the new key hash and each still fetches its original value on a task. - AC-6.5. While a rotation is in progress, tasks can still fetch every credential, whether it is on the old key or the new one. *Verify:* run a task fetching a not-yet-rekeyed credential during rotation, which succeeds. - AC-6.6. Starting rotation while an operation runs ⇒ 409 "A NodeVault rotation is already running" with a link. - AC-6.7 (D8). Retire old key: Settings → NodeVault lists each old key with the number of credentials still on it. "Retire" is enabled only when that count is 0. It removes the key from every node (offline nodes on reconnect), is audited, and can't be undone. *Verify:* during rotation, Retire is refused ("3 credentials still use this key"). After rekey it is allowed; afterwards `jaah nodevault keys` on each node no longer lists that key. - AC-7.1 (rotation distribution, rekey, retirement) durable with deterministic ids, restart-safe, no duplicate side effects. - AC-1.3 (part): rotation and retiring refused with "NodeVault isn't included in your plan" when frozen. - AC-9.1 (part): credential rekeyed (old → new hash), old key retired, key generated/uploaded + backup confirmed at rotation. Out of scope: Do NOT change the setup/delivery/migration workflow semantics except to call into them; do NOT build web UI; no "compromise rotation" per-node wrap (F7, deferred). Epic out of scope (PRD §6, verbatim): Don't change encryption or consumption for CP-consumed credentials (forge/git, Jira, Slack, notify targets, cloud-launch AWS, org shared store, LLM, Bedrock, Stripe, embeddings, headscale, MFA stay on ADR-0137 KMS; consumers aren't re-plumbed to nodes). Don't touch tier 1 (CREDENTIAL_ENCRYPTION_KEY). Don't change the node-token mechanism (`node_credentials.py`, /etc/skynet/node.env), the login-seal keys (ADR-0133), or the `jaah-credential` broker names for git, GitHub, Bedrock and account credentials. No OAuth2 in NodeVault; don't alter `oauth2_tokens.py`. Don't build anything in 5246's area beyond the one NodeVault plan parameter. No hosted-node hardening and no "Jaah-hosted" label. No node-side Test button, node-side project env resolution, transcript redaction, BYOK/XKS or external secret references (F1–F8 deferred). No Jaah-held escrow of any NodeVault private key or backup. No logging of any secret value, private key, wrapped-key blob contents or backup passphrase anywhere (logs, activity events, DBOS, Slack, task summaries). No admin notification on key delivery (AC-9.3 dropped, E4). Done means: - Workflow tests: rotate happy path; promote waits only on online capable nodes; offline node catches up by chain; rekey idempotent across a simulated crash (all 5 on new hash, rerun changes nothing); seal racing promotion is swept; retire refused with credentials left (exact text with count); retire completes only when every grant retired; offline node keeps op retiring; retire abandon restores previous — cite. - AC-6.5: during a rotation on a local stack, a task fetches a not-yet-rekeyed credential successfully — cite. - AC-6.4 executed on a local stack with kill mid-way; AC-6.7 `jaah nodevault keys` after retire — cite observed output. - `./dc check full` green; PR merged into develop through the security review lane / approval; rendered report cites file:line per AC and the observed output of every oracle above. Required checks: test-stack

NodeVault rotation, verified rekey, retire and abandon merged (#4706, 957d4ae1c) after security approval. Full gate green; real dev-node E2E (API killed mid-rekey, retire, keys listing) passed. Deferred: authenticate rotation vs an active CP (outside D9).