CP NodeVault crypto seam and schema

## Original requirement (epic 5269, verbatim) Build NodeVault feature: - Is a product that can be activated on a billing plan (once it lands in 5246) - When NodeVault is activated for an org, Org Admin can start Setup in Settings / NodeVault, that initiates a stepper: - Explanation with a diagram - On the backend System generates a Temporary NodeVault Key and sends it to all nodes (encoding each with node key) in the organization. Nodes store it in a secure place (next to other sensitive information, such as node token) - Generate NodeVault Key / Upload NodeVault Key. When Generate NodeVault Key - browser generates a key-pair (same algo as Temporary) encodes it with Temporary NodeVault Key public key and sends to CP to fan out to all nodes. Stores public key in the DB - Migrate credentials to NodeVault: all, select. Backend goes through all credentials that user selected and that are KMS encoded, decodes them and re-encodes them with NodeVault Key public key. No cred logging. - Show the result to the user - Credentials get a checkbox NodeVault (on by default for each new cred) - Credentials remember their current NodeVault key hash - When a broker requests a credentials and the credential is NodeVault, the broker uses the private NodeVault Key to decode it - Admin can rotate the NodeVault in Settings / NodeVault. The rotation procedure is a stepper: - Generate / Upload - same as at Setup - Migrate credentials: some of the creds may still be KMS - offer to migrate them - The new key is encrypted with current key (in browser) and fanned out to all nodes. Nodes decode and store the new key, keeping the old one - CP goes through each key and sends them to nodes to rekey. Nodes decode the cred, reencode it with new public and send them back to CP - All these workflows must be resilient and retry when something fails. Use DBOS to make it as reliable and idempotent as it gets Goal: reliable, provable way of Jaah not seeing and not being able to obtain access to client's credentials with all paths hermetic and auditable. ## Settled inputs (binding — realize, do not reopen) Full documents: BA PRD = Task 5273's `prd` artifact; Architecture = Task 5274's `architecture` artifact (`task_read` 5273 / 5274, then `task_read_artifact`). READ the architecture sections this child names — §2 crypto constructions, §3.2 data model, §4 interface contracts are PINNED and every child builds against them exactly. Binding decisions: D1 node-only (CP-consumed creds stay KMS) · D2 per-node identity key pairs (NIK) replace the CP temp key · D3 Jaah-hosted nodes included · D4 mandatory passphrase backup, no Jaah escrow · D5 later-joining nodes get the key automatically from a peer · D6 NodeVault is a plan parameter with per-org override · D7 OAuth2 excluded · D8 old keys kept until admin retires (refused while any credential uses it) · D9 passive-breach-proof + audited, no claim vs malicious Jaah · D11 plan off after setup ⇒ existing NV creds keep working, setup/rotation/migration/retire/new-NV frozen · D12 Test disabled for NV · D13 rogue-node enrollment is detectable (audit + node list), not impossible · E1 vendor `cryptography` into jaahd · E2 rotation chain-wraps new key under CURRENT key (K1: rotation does not recover from a leaked current key) · E3 DBOS for every workflow (carve-out from ADR-0074) · E4 AC-9.3 dropped (no admin notification on key delivery) · E5 node key files daemon-owned 0600 in a 0700 dir. Pinned crypto (arch §2): RSA-4096 e=65537 RSAES-OAEP SHA-256/MGF1-SHA-256 everywhere; AES-256-GCM 12-byte nonce; key hash = hex(sha256(SPKI DER)); envelope key id `nv:<hash>`; secret envelope = ADR-0137 v2 tier-2 unchanged; key blob `{v:1,k,alg,ek,n,ct}` AAD `nodevault.key|v1|<org_id>|<key_hash>|<recipient>`; backup PBKDF2-SHA256 600k. SECURITY SURFACE: this epic is credentials/crypto. Never log/checkpoint/return a secret value, private key, data key, passphrase or backup. The merge needs the security review lane or human approval per branching policy. Quality bar: "technically passes" is the floor. A build that satisfies the words but leaks key material to the CP, marks a step done before the node acknowledged it, or shows raw ids/stack traces to the user FAILS this child. ## This child — scope (HOW) Architecture child C2 is the node side; this is architecture child C3 (arch §3.2, §3.3, §4.2 identity route). - ONE additive Alembic migration (apps/api/migrations/versions) for every §3.2 table/column EXCEPT the plan-parameter columns (separate child, which chains after this migration): `nodes.nodevault_identity_key/_fp/_capable/_identity_at`; `nodevault_keys` (EntityMixin+OrgScoped, state pending|current|previous|retiring|retired|abandoned, source, chain_blob, backup_confirmed_at, retired_at, partial unique one `current` per org); `nodevault_grants` (unique node_id,key_hash; blob, blob_kind, state waiting|delivered|failed|retire_pending|retired, last_error closed vocab, generation, delivered_at); `nodevault_operations` (kind, state, step, key_hash, prev_key_hash, counts, workflow_id; partial unique (org_id) WHERE state='running'); `nodevault_operation_items` (phase, state, reason, staged ciphertext-only JSONB); `credentials.nodevault` bool default false. Models in models.py. - `credential_keys.NodeVaultPublicKeyProvider(key_hash, spki)`: wrap works, `unwrap_data_key` raises `NodeVaultSealed`; `provider_for` dispatches `nv:` to the org's `nodevault_keys` row (lookup takes (session, org_id); thread it from callers). - `credential_crypto`: `open_sealed`/`open_secret` raise typed `NodeVaultSealed` for `nv:`; `sealed_secret_from_wire` admits a `k` naming the org's `current` or `previous` NV key only when the posted credential has nodevault=true, else KMS only; invariant `nodevault ⇔ every tier-2 envelope k starts with "nv:"` checked over the FINAL merged config of every write (PATCH kept fields included); `reseal_secret_to_nodevault(row, field, provider)` (KMS open + NV seal in one call, plaintext a local only); `reseal_to_current_keys` skips `nv:`. A derived `nodevault_key_hash` for listing (never stored twice). - `POST /api/agent/nodevault/identity {public_key, capable}` (X-Agent-Token): CP computes fingerprint; same key re-posted = 200 idempotent; a different key = 409 "identity key already registered"; `capable` may change each post; after commit calls `ensure_grants(node)`. - `nodevault_grants.py`: `ensure_grants(session, node_id)` and `start_grant_workflow(grant)` exactly per arch §3.3 (set-up = org has a `current` key or a running setup op; waiting grants for every live key a capable node lacks; retire_pending exempt from setup/capable preconditions; DBOS status check PENDING/ENQUEUED ⇒ nothing, absent ⇒ start, finished-unsettled ⇒ generation+1 in same txn). Workflow ids `nodevault-<deliver|retire-node>:{node}:{key_hash}:{generation}`; the workflow BODIES belong to the delivery-workflows child — start by name and test against a DBOS fake. ## Acceptance criteria carried by this child (PRD, verbatim unless marked amended) - AC-8.1. No CP path opens a NodeVault secret. The CP has no code path that can decrypt a NodeVault envelope. `open_secret` (and every successor) refuses a NodeVault envelope with a typed error, and the CP holds no NodeVault private key. *Verify:* a test calling `open_secret` on a NodeVault envelope gets the refusal. Search the CP codebase and DB for NodeVault private key storage and find none. An independent security-review lane signs this off (security-surface change). - AC-4.6. Key hash. The NodeVault key is identified everywhere by a hash of its public key (SHA-256 of SPKI DER, shown shortened). *Verify:* the console's key hash equals `openssl pkey -pubin -outform DER | sha256sum` over the stored public key. - AC-5.5 (A9, stored half). Each NodeVault secret envelope records the key hash it is sealed under, and the credential list and detail show it. *Verify:* the DB envelope `k` (or its successor) equals the key hash (console display lands with the credential-form child). - AC-3.1 (CP half): the public half and its fingerprint are registered on the node's CP row. Out of scope: Do NOT build the plan parameter columns or nodevault_policy (plan child), the DBOS workflow bodies or /api/nodevault routes (workflow children), the runner fetch NV branch (runner fetch child), or credential_consumers.py (eligibility child). Do NOT alter KMS behaviour for non-`nv:` envelopes. Epic out of scope (PRD §6, verbatim): Don't change encryption or consumption for CP-consumed credentials (forge/git, Jira, Slack, notify targets, cloud-launch AWS, org shared store, LLM, Bedrock, Stripe, embeddings, headscale, MFA stay on ADR-0137 KMS; consumers aren't re-plumbed to nodes). Don't touch tier 1 (CREDENTIAL_ENCRYPTION_KEY). Don't change the node-token mechanism (`node_credentials.py`, /etc/skynet/node.env), the login-seal keys (ADR-0133), or the `jaah-credential` broker names for git, GitHub, Bedrock and account credentials. No OAuth2 in NodeVault; don't alter `oauth2_tokens.py`. Don't build anything in 5246's area beyond the one NodeVault plan parameter. No hosted-node hardening and no "Jaah-hosted" label. No node-side Test button, node-side project env resolution, transcript redaction, BYOK/XKS or external secret references (F1–F8 deferred). No Jaah-held escrow of any NodeVault private key or backup. No logging of any secret value, private key, wrapped-key blob contents or backup passphrase anywhere (logs, activity events, DBOS, Slack, task summaries). No admin notification on key delivery (AC-9.3 dropped, E4). Done means: - Test: open_secret/open_sealed on an nv: envelope raises NodeVaultSealed (never plaintext, never 500) — cite. - Test: wire admission accepts nv:<current>/nv:<previous> only with nodevault=true; rejects nv:<retiring|abandoned|unknown>; PATCH whose merged config mixes kms and nv envelopes is a 422 — cite. - Test: key hash computed by the CP equals `openssl pkey -pubin -outform DER | sha256sum` over a fixture key — cite command + output. - Test: identity POST idempotent for same key (200), 409 for a different key, fingerprint CP-computed — cite. - Test: ensure_grants/start_grant_workflow against a DBOS fake: start-if-absent, no-op when PENDING, generation bump when finished-unsettled, retire_pending restarted regardless of capable — cite. - Migration upgrade, downgrade, upgrade clean on a local stack; `grep -rn "BEGIN PRIVATE KEY" apps/api/app` finds no NV private key handling — cite. - `./dc check full` green; PR merged into develop through the security review lane / approval; rendered report cites file:line per AC and the observed output of every oracle above. Required checks: test-stack

NodeVault CP seam + schema (migration 0377, wrap-only provider, NodeVaultSealed refusal, nv: admission + invariant, identity route, ensure_grants) merged via #4626 (human-approved); #4650 fixed the API-ref spec. Full gate green.