Build NodeVault Credential Encryption Feature

Task 5269 · Done · Epic · created · updated

Build NodeVault feature: - Is a product that can be activated on a billing plan (once it lands in 5246) - When NodeVault is activated for an org, Org Admin can start Setup in Settings / NodeVault, that initiates a stepper: - Explanation with a diagram - On the backend System generates a Temporary NodeVault Key and sends it to all nodes (encoding each with node key) in the organization. Nodes store it in a secure place (next to other sensitive information, such as node token) - Generate NodeVault Key / Upload NodeVault Key. When Generate NodeVault Key - browser generates a key-pair (same algo as Temporary) encodes it with Temporary NodeVault Key public key and sends to CP to fan out to all nodes. Stores public key in the DB - Migrate credentials to NodeVault: all, select. Backend goes through all credentials that user selected and that are KMS encoded, decodes them and re-encodes them with NodeVault Key public key. No cred logging. - Show the result to the user - Credentials get a checkbox NodeVault (on by default for each new cred) - Credentials remember their current NodeVault key hash - When a broker requests a credentials and the credential is NodeVault, the broker uses the private NodeVault Key to decode it - Admin can rotate the NodeVault in Settings / NodeVault. The rotation procedure is a stepper: - Generate / Upload - same as at Setup - Migrate credentials: some of the creds may still be KMS - offer to migrate them - The new key is encrypted with current key (in browser) and fanned out to all nodes. Nodes decode and store the new key, keeping the old one - CP goes through each key and sends them to nodes to rekey. Nodes decode the cred, reencode it with new public and send them back to CP - All these workflows must be resilient and retry when something fails. Use DBOS to make it as reliable and idempotent as it gets Goal: reliable, provable way of Jaah not seeing and not being able to obtain access to client's credentials with all paths hermetic and auditable. In BA and Design phases - stress test the idea and do a deep analysis and research if that is actually the best practice or suggest more robust ideas.

Subtasks

  • [business_analysis] Build NodeVault Credential Encryption Feature Done
    NodeVault PRD: 18 asks in scope, 13 decisions (12 from the operator interview, 1 assumed), 41 ACs with oracles. Temp key replaced by per-node keypairs; node-only (CP-consumed and OAuth2 creds excluded); mandatory backup; admin retires old keys. Depends on 5246. Assumed: D13.
  • [architecture] Build NodeVault Credential Encryption Feature Done
    NodeVault = 2nd key provider on the ADR-0137 envelope (k=nv:<hash>), opened only by node daemons. Vendored cryptography, chain-wrap rotation, DBOS throughout. 9 children C1–C9. Operator answers E1–E5 (AC-9.3 dropped, AC-3.1 amended).
  • [plan] Build NodeVault Credential Encryption Feature Done
    Split NodeVault by seam (node/CP/workflows/web) into 14 coders across 7 waves (5283–5296); arch C5 split 3 ways, C8 4 ways. Plan child 5285 waits on 5257 (plan_terms), not yet merged. AC-9.3 dropped (E4).
  • CP NodeVault crypto seam and schema Done
    NodeVault CP seam + schema (migration 0377, wrap-only provider, NodeVaultSealed refusal, nv: admission + invariant, identity route, ensure_grants) merged via #4626 (human-approved); #4650 fixed the API-ref spec. Full gate green.
  • NodeVault plan parameter and policy Done
    NodeVault plan parameter (default off, per-org override, migration 0381), toggles in plan editor + org Plan tab, nodevault_policy.status gate, boolean-only audit. Full gate green; security-approved; PR #4662 merged. Dev-deploy check pending.
  • Node NodeVault command handlers Done
    Built the six NodeVault node command handlers (install, retire, held_keys, wrap_for_node, rekey, verify) with vector tests. Merged in #4663 after operator security approval. CI gate green. Local full gate was killed by host memory pressure, so per the operator's call CI gated the PR.
  • NodeVault runner credential fetch path Done
    NodeVault runner fetch merged (#4665, 9756aac79). The CP serves NV fields sealed (envelope + AAD), records node_id and calls ensure_grants on a miss; the node broker opens them and returns nodevault_key_missing if it lacks the key. Full gate green, local E2E passed, security review passed, operator approved the merge.
  • Browser NodeVault key crypto module Done
    nodeVaultKeys.ts (WebCrypto NVK gen, node/chain blobs, PBKDF2 backup) + 2-way Python interop test merged via #4664 (security gate human-approved, CI gate green). Local full rerun OOM-reaped; operator chose CI as gate. UI caller-contract follow-ups in report.
  • NodeVault credential eligibility and consumers Done
    NodeVault eligibility and consumers merged (#4672, c8ac66e03): consumer registry, a refusal on every write path, OAuth2 types ineligible, env vars unresolved, Test refused, plan freeze, uncheck rule, audit. Full gate and CI green; operator-approved. Deferred: row lock (L3).
  • NodeVault key delivery workflows and status API Done
    NodeVault key-delivery + retire-node DBOS workflows, hello wake-ups and /api/nodevault routes merged (#4681, 42f54e909) after human security approval. Full gate green; local E2E (real node crypto, API killed mid-run) passed; review findings fixed.
  • NodeVault setup and migration workflows Done
    NodeVault setup + migration workflows and routes merged (#4688, 6f5b8cef6) after operator security approval. Full gate and CI green; real dev-node E2E (AC-4.7, API killed mid-run) passed. Deferred: consumer-door row lock (C7 L3), retire generation re-check.
  • NodeVault credential form and node fingerprints Done
    NodeVault checkbox and in-browser sealing on the credential form, key hash in list and detail, node identity fingerprints, Audit NodeVault filter. Full gate green; human security approval; #4685 merged. Deferred: frozen-plan seal-key vs editing NV secrets.
  • NodeVault rotation, rekey and retire workflows Done
    NodeVault rotation, verified rekey, retire and abandon merged (#4706, 957d4ae1c) after security approval. Full gate green; real dev-node E2E (API killed mid-rekey, retire, keys listing) passed. Deferred: authenticate rotation vs an active CP (outside D9).
  • Settings NodeVault page and setup stepper Done
    Settings → NodeVault page + 6-step Setup stepper (server-owned step, browser-only key, mandatory backup, 422 resubmit). Full gate green; review lanes pass; human security approval; #4696 merged (92e31e660). Deferred: re-confirm recipients on node-set change.
  • NodeVault rotation stepper and retire UI Done
    NodeVault rotation stepper + Retire UI merged (#4740, d27e092e3); Jaah Deploy green. e2e 1280/400 + 19 unit pass, review lanes pass. Full gate stopped by host memory, merged on operator's "Merge now". Deferred: stop-rekey action.
  • NodeVault ADR, proofs and dev E2E Done
    ADR-0154 + compliance note + AC-8.2/AC-7.2 proof tests merged (#4756, dec8d12ef); Jaah Deploy green; review lanes pass. Open (operator decision): dev E2E AC-10.1/5.7/1.3 — runner can't reach dev.jaah.ai. Deferred: peer-blob shape check.
  • [finish] Build NodeVault Credential Encryption Feature Done
    All 14 NodeVault coders done, 15 PRs merged, dev deploy green; code matches the ask. Gap: dev E2E (AC-10.1/5.7/1.3) never run — no runner reaches dev.jaah.ai — filed as Task 5480.
  • NodeVault dev E2E on dev.jaah.ai (AC-10.1, AC-5.7 Done
    NodeVault dev E2E run on dev.jaah.ai (Org B). AC-10.1, AC-5.7 and AC-1.3 all pass with screenshots. No NodeVault defect, no code change. Non-NodeVault dispatch, migration and dev-env findings are named in the report.

Artifacts

Activity

  • Created
  • Runner started
  • Ready
  • Ready
  • Ready
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Runner ended (done)
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Runner started
  • Runner started
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner ended (done)
  • Runner ended (done)
  • Runner started
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Runner ended (done)
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Ready
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Ready
  • Runner ended (done)
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner ended (done)