NodeVault credential eligibility and consumers

## Original requirement (epic 5269, verbatim) Build NodeVault feature: - Is a product that can be activated on a billing plan (once it lands in 5246) - When NodeVault is activated for an org, Org Admin can start Setup in Settings / NodeVault, that initiates a stepper: - Explanation with a diagram - On the backend System generates a Temporary NodeVault Key and sends it to all nodes (encoding each with node key) in the organization. Nodes store it in a secure place (next to other sensitive information, such as node token) - Generate NodeVault Key / Upload NodeVault Key. When Generate NodeVault Key - browser generates a key-pair (same algo as Temporary) encodes it with Temporary NodeVault Key public key and sends to CP to fan out to all nodes. Stores public key in the DB - Migrate credentials to NodeVault: all, select. Backend goes through all credentials that user selected and that are KMS encoded, decodes them and re-encodes them with NodeVault Key public key. No cred logging. - Show the result to the user - Credentials get a checkbox NodeVault (on by default for each new cred) - Credentials remember their current NodeVault key hash - When a broker requests a credentials and the credential is NodeVault, the broker uses the private NodeVault Key to decode it - Admin can rotate the NodeVault in Settings / NodeVault. The rotation procedure is a stepper: - Generate / Upload - same as at Setup - Migrate credentials: some of the creds may still be KMS - offer to migrate them - The new key is encrypted with current key (in browser) and fanned out to all nodes. Nodes decode and store the new key, keeping the old one - CP goes through each key and sends them to nodes to rekey. Nodes decode the cred, reencode it with new public and send them back to CP - All these workflows must be resilient and retry when something fails. Use DBOS to make it as reliable and idempotent as it gets Goal: reliable, provable way of Jaah not seeing and not being able to obtain access to client's credentials with all paths hermetic and auditable. ## Settled inputs (binding — realize, do not reopen) Full documents: BA PRD = Task 5273's `prd` artifact; Architecture = Task 5274's `architecture` artifact (`task_read` 5273 / 5274, then `task_read_artifact`). READ the architecture sections this child names — §2 crypto constructions, §3.2 data model, §4 interface contracts are PINNED and every child builds against them exactly. Binding decisions: D1 node-only (CP-consumed creds stay KMS) · D2 per-node identity key pairs (NIK) replace the CP temp key · D3 Jaah-hosted nodes included · D4 mandatory passphrase backup, no Jaah escrow · D5 later-joining nodes get the key automatically from a peer · D6 NodeVault is a plan parameter with per-org override · D7 OAuth2 excluded · D8 old keys kept until admin retires (refused while any credential uses it) · D9 passive-breach-proof + audited, no claim vs malicious Jaah · D11 plan off after setup ⇒ existing NV creds keep working, setup/rotation/migration/retire/new-NV frozen · D12 Test disabled for NV · D13 rogue-node enrollment is detectable (audit + node list), not impossible · E1 vendor `cryptography` into jaahd · E2 rotation chain-wraps new key under CURRENT key (K1: rotation does not recover from a leaked current key) · E3 DBOS for every workflow (carve-out from ADR-0074) · E4 AC-9.3 dropped (no admin notification on key delivery) · E5 node key files daemon-owned 0600 in a 0700 dir. Pinned crypto (arch §2): RSA-4096 e=65537 RSAES-OAEP SHA-256/MGF1-SHA-256 everywhere; AES-256-GCM 12-byte nonce; key hash = hex(sha256(SPKI DER)); envelope key id `nv:<hash>`; secret envelope = ADR-0137 v2 tier-2 unchanged; key blob `{v:1,k,alg,ek,n,ct}` AAD `nodevault.key|v1|<org_id>|<key_hash>|<recipient>`; backup PBKDF2-SHA256 600k. SECURITY SURFACE: this epic is credentials/crypto. Never log/checkpoint/return a secret value, private key, data key, passphrase or backup. The merge needs the security review lane or human approval per branching policy. Quality bar: "technically passes" is the floor. A build that satisfies the words but leaks key material to the CP, marks a step done before the node acknowledged it, or shows raw ids/stack traces to the user FAILS this child. ## This child — scope (HOW) Architecture child C7 (arch §3.3 credential_consumers / eligibility / tester / uncheck, §4.1 credential-form endpoints). - `apps/api/app/credential_consumers.py` (new single home for D1): declarative registry of every CP-side consumer `(label, FK column or JSONB/settings path, user-facing reason)`. Seed from the FK census (models.py project_credentials, projects, accounts, cloud_node_defaults, node_groups, gha_runner_pools, org_shared_stores, user_git_connections, admin_users) PLUS JSONB/settings references you must enumerate: notify targets, Slack socket, Jira, LLM, Bedrock, Stripe, support_kb, headscale, MFA, site_enquiry, gmind_tenancy, tenancy offboard, vnodes (PRD AC-5.3 list). `consumers_of(session, credential_id)`, `refuse_if_nodevault(credential, consumer_label)` called by every consumer's write route; at consumption raise `NodeVaultUnavailable` ("NodeVault credential — not available to Jaah"), never a 500. A census test fails if a new FK to credentials is added without a registry row. - `credentials/registry.py` CREDENTIAL_TYPES gains `nodevault_eligible` (False for every OAuth2 type). Eligible = type eligible AND consumers_of empty. - `project_env.py` resolver: a reference to an NV field resolves as unavailable with reason "NodeVault" and is omitted from the runner env. - `/api/credentials/test` refuses NV with the AC-5.9 reason. - Credential create/PATCH: NV create or toggling NV on refused when `nodevault_policy.status` is not allowed/set up (D11 freeze, "NodeVault isn't included in your plan"); unchecking requires a full new KMS-sealed tier 2 (posted nodevault=false with any kept nv: envelope ⇒ 422); toggling records "NodeVault toggled on a credential" audit event. - Endpoints: `GET /api/credentials/nodevault-state → {state:"not_setup"|"active"|"frozen"|"not_in_plan", current_key_hash|null, types:{<slug>:{eligible,reason|null}}}`; `GET /api/credentials/{id}/nodevault-eligibility → {eligible, reason|null, consumers:[label]}` (credentials:view). ## Acceptance criteria carried by this child (PRD, verbatim unless marked amended) - AC-5.3. Eligibility (D1). A credential is NodeVault-eligible only if no CP-side consumer opens it. Two rules follow: Marking an in-use credential NodeVault is refused with the consumer named. Pointing a CP consumer at a NodeVault credential is refused, or that consumer reports "NodeVault credential — not available to Jaah". It never fails with a 500. *Verify:* try to pick a NodeVault credential as a project's forge or Slack credential and see a clear refusal. Try to check NodeVault on a credential already wired as a notify target and see a refusal naming it. - AC-5.4. Project env vars. A project env var that references a NodeVault credential field resolves **as unavailable with the reason "NodeVault"** on the Environment tab, and is omitted from the runner env, consistent with R9 "unresolved reference omits the line". *Verify:* add such a reference and check that the Environment tab shows the reason and the runner env lacks the variable. - AC-5.8 (API). Unchecking NodeVault on an existing credential requires **re-entering every secret value**, which is then sealed under KMS in the browser, because no path may hand a NodeVault plaintext back to the CP. *Verify (API half):* a PATCH with nodevault=false keeping an nv: envelope is refused; a full KMS tier 2 succeeds and the envelope names the KMS key. - AC-5.9 (D12, API). Test is disabled for NodeVault credentials with the reason "Testing runs on Jaah's servers, which can't open NodeVault credentials." - AC-1.3 (part): creating new NodeVault credentials is refused with "NodeVault isn't included in your plan" when frozen. - AC-9.1 (part): "NodeVault toggled on a credential" event. Out of scope: Do NOT re-plumb any CP consumer to nodes or change how they open KMS credentials; do NOT build the credential form UI (credential-form child); do NOT implement node-side env resolution (F4). Epic out of scope (PRD §6, verbatim): Don't change encryption or consumption for CP-consumed credentials (forge/git, Jira, Slack, notify targets, cloud-launch AWS, org shared store, LLM, Bedrock, Stripe, embeddings, headscale, MFA stay on ADR-0137 KMS; consumers aren't re-plumbed to nodes). Don't touch tier 1 (CREDENTIAL_ENCRYPTION_KEY). Don't change the node-token mechanism (`node_credentials.py`, /etc/skynet/node.env), the login-seal keys (ADR-0133), or the `jaah-credential` broker names for git, GitHub, Bedrock and account credentials. No OAuth2 in NodeVault; don't alter `oauth2_tokens.py`. Don't build anything in 5246's area beyond the one NodeVault plan parameter. No hosted-node hardening and no "Jaah-hosted" label. No node-side Test button, node-side project env resolution, transcript redaction, BYOK/XKS or external secret references (F1–F8 deferred). No Jaah-held escrow of any NodeVault private key or backup. No logging of any secret value, private key, wrapped-key blob contents or backup passphrase anywhere (logs, activity events, DBOS, Slack, task summaries). No admin notification on key delivery (AC-9.3 dropped, E4). Done means: - Tests: consumers_of finds each registry entry type (FK + JSONB); toggling NV on an in-use credential ⇒ refusal naming the consumer; pointing forge/Slack/notify at an NV credential ⇒ clear 4xx; consuming one ⇒ NodeVaultUnavailable not 500; census test fails on an unregistered FK — cite. - Tests: OAuth2 type ineligible with reason "OAuth2 credentials can't use NodeVault"; project_env NV reference resolves unavailable "NodeVault" and absent from runner env; tester refusal exact text; uncheck 422; frozen create 403 exact text; toggle audit event — cite. - `./dc check full` green; PR merged into develop through the security review lane / approval; rendered report cites file:line per AC and the observed output of every oracle above. Required checks: test-stack

NodeVault eligibility and consumers merged (#4672, c8ac66e03): consumer registry, a refusal on every write path, OAuth2 types ineligible, env vars unresolved, Test refused, plan freeze, uncheck rule, audit. Full gate and CI green; operator-approved. Deferred: row lock (L3).