Node and Multi-Node Host transfer

Task 5697 · Done · Coder · created · updated

## Epic requirement (verbatim) > Allow Operator to change ownership of everything. If he changes the ownership of a project - the project and all tasks and everything related moves to another org. > > Operator can see everything. He has a global org filter. Make it a state of the art method of managing - do a deep web research Full settled specs: the PRD (task #5681) and the Architecture (task #5682) of epic #5680 — read them with `task_read` / `task_read_artifact`. This body carries the parts that bind THIS child verbatim. ## Requirements this child serves (PRD R3, verbatim) - R3.1 "An operator can change a node's org. The DB trigger forbidding it is replaced by a rule that admits exactly this operator action (every other write path still refused). ADR-0118 is amended in place to say so." - R3.2 "Old-org worktrees on the node (Q6: release their slots): every slot of an org-A project on the node is released (committed work on origin stays; uncommitted discarded, listed per slot) and the node's org-A project placements and node-group membership are removed." - R3.3 "A live org-A runner on the node BLOCKS the move (listed as a blocker with a link to stop it)." - R3.4 "After the move the node holds no org-A secret: org-A runner-account credential dirs and NodeVault grants for org A are removed from it, and it runs only org-B tasks." - R3.5 "A Multi-Node Host moves with all its co-located (virtual) nodes as one action; R3.2–R3.4 apply to each." - Offline node (PRD §4 states): "offline: move allowed, secrets removal happens on its next contact". ## Approach (Architecture C-D, verbatim) - **Trigger** (`tenancy/node_org.py`). The function admits a change only when `OLD.id::text = ANY(string_to_array(current_setting('jaah.node_org_transfer', true), ','))`. The engine sets it with `SELECT set_config('jaah.node_org_transfer', :ids, true)` (transaction-local; `SET LOCAL` takes no bind params) right before `rehome_set`, and resets it to `''` right after. Everything else still raises. `ENABLE ALWAYS` is kept. A census test asserts that the setting name appears in exactly one function, in `transfer.py`. Migration re-creates the function (this child owns the ONE migration of its wave). ADR-0118 is amended in place ("a node changes org only through the operator transfer"). - **Node rule.** Every `Derived(Node)` child is named, since following by key would stamp org-A rows B: - `Erase` (rows deleted, payload intents): worktrees of org-A projects (D2 semantics), slot_intents, env_intents, gha_runner_pools, node_projects (`Drop`). - `Drop`: nodevault_grants (deleted, and the org-A key's grant retired through `nodevault_grants`/`durable/nodevault.py` retire-node workflow); account_observations; credential_oauth_flows of org-A credentials; console_api_key_leases. - `Follow`: vnode_hosts, node_host_metrics. - `Detach`: `nodes.node_group_capacity_id` → NULL (leaves A's group); `accounts.node_id` for org-A accounts pinned here (the account stays in A, unpinned, listed). - `Stays`: command_audit_log, alerts, run_telemetry (provenance). - `Erase` (node disk): org-A runner-account dirs and NodeLink leaves, reusing `offboard.delete_account_dirs` / `offboard.delete_shared_leaves` (no copy). - `Block`: a live org-A runner on the node (R3.3), with a stop link. - **Offline node**: the intents are durable and apply on reconnect. The manifest note reads "cleanup pending until <node> reconnects". - **After commit**: `nodelink.request_converge` for the new org's store. - **Host** (kind `host`): the root is the host node. Every co-located node (`virtual_nodes.node_id`) joins the same transaction; each is planned by the node rule and named in the setting's id list. One manifest. Register kinds `node` and `host` on the engine via `transfer.register(kind, KindRule)` (contract from the engine child; do not change it). The satisfiability/totality test must cover both roots. ## Acceptance criteria (PRD, verbatim) - AC9 "Node move: node shows org B; A's slots on it released and listed; node no longer in A's node group; it runs B's tasks and refuses A's; no A account dir/NodeVault grant remains on it. A live A runner on it blocks the move with a stop link." - AC10 "Host move moves all its virtual nodes; each satisfies AC9." - AC14 (node) "Failure is all-or-nothing: a blocked/failed move leaves every row in A and the UI states why." (live runner on node → nothing moved) - Plus: any write to `nodes.org_id` outside the engine still raises (test via a direct UPDATE and via an ordinary node edit route). Credentials & reach (Architect §8, verbatim rows: oracle | credential/identity | status): - AC4 and unit/integration tests | worktree only (`./dc check full`) | n/a - AC5/AC9/AC10 need org-B capacity and a Multi-Node Host | dev inventory via the operator login | **verified**: Org B owns 2 nodes; Gmind 3, Wealthcharts 3, TimeSaver2 1, Operator 1; `/api/vnode-hosts` lists 3 hosts - AC9 "no A account dir / NodeVault grant remains on the node" (on-disk part) | no console surface shows a node's on-disk account dirs | **deferred** (the hub forbids non-UI inspection of live state): the oracle is an API test asserting the grants are retired and the erase intents are written and acknowledged. The DB half (grants, accounts) is checked in the Nodes UI Out of scope: - Do NOT allow any other path to change `nodes.org_id`; do NOT drop `ENABLE ALWAYS`. - Do NOT copy or rewrite offboarding logic — call `offboard.delete_account_dirs` / `delete_shared_leaves`. - Do NOT build the dialog UI (the dialog child) or the standalone object kinds (the objects child). - Do NOT carry org-A secrets, grants or account dirs into org B. PRD §5 (verbatim, binds every child): - Do NOT add user/membership moves (Q1 not selected) or change `admin_users.org_id` writes. - Do NOT change the account-menu org switcher (`components/account/OrgSwitcherList.tsx`) or `/api/auth/switch` semantics, or the stale-tab guard (`lib/activeOrg.ts`, `stale_tab_guard.py`). - Do NOT add impersonation / "act as org". - Do NOT add plan-limit checks to moves; do NOT touch billing ledgers' org (R2.2) or Stripe objects. - Do NOT weaken tenant read narrowing for non-operators (`authz.py::_bind_read_scope`); the filter only narrows an operator further. - Do NOT allow any path other than the operator move to change `nodes.org_id`. - Do NOT carry a shared dependency or a secret silently into org B; do NOT copy credentials. - Do NOT add email notifications, bulk move, or scheduled moves. - Do NOT move offboarding/deletion logic (`tenancy/offboard.py`, `org_delete.py`) — reuse, don't rewrite. - Nothing ships behind a flag or setting (R6/AC20: "Everything above is ON by default for operators on the dev control plane (dev.jaah.ai) with no flag; no \"ship disabled\".") Done means: - AC9 — `apps/api/test_org_transfer_node.py`: idle node move → node org B, A slots erased + intents written and listed in manifest, node detached from A's group, scheduler places B task and never A task, nodevault grants retired and account-dir/NodeLink erase intents written (on-disk part: deferred — the hub forbids non-UI inspection of live state; this API test is the oracle). Live A runner → 409 `blocked` with stop link, nothing moved. - AC10 — same file: host move moves every virtual node in one transaction; each satisfies AC9. - AC14 (node) — blocked move leaves all rows unchanged (snapshot test). - Trigger: direct UPDATE of `nodes.org_id` without the setting raises; census test asserts the setting name appears only in `transfer.py`. - ADR-0118 amended in place; `python3 tools/check_context.py` green. - `./dc check full` green.

Outcome

Node/host org transfer shipped: setting-gated trigger, slot release, grant retire, durable account-dir erase + placement fence. Full gate green, PR #4982 merged. Deploy queued behind a hung runner (operator: settle now). mem-2 used.

Pull request

Artifacts

Activity

  • Draft
  • Ready
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)