Co-located nodes don't see general NodeLink
## Defect
On co-located nodes (vnodes), general NodeLinks are never usable by runners. This is the epic's main purpose (a swarm of agents on a shared file system, epic 5686), and the defect blocks it on every Multi-Node Host.
**What was observed** on app.dev.jaah.ai, Gmind, Task 5734 (report artifact 4161):
- The vnodes were aws1y, pool-b, and swarm-vn1 (born from the Add-node wizard with Accounts and swarm-a ticked).
- The host helper reported general NodeLinks bound with no error.
- The vnode's own daemon reported `not_mounted` for each of them in every snapshot.
- The newborn swarm-vn1 did see its legacy `shared/accounts` EFS mount, which is made before its units start. It saw none of the helper's later binds under `home/.skynet`.
- Plain nodes are not affected, because their jaahd shares the host mount namespace.
## Cause (static recon, not yet confirmed live)
- `packaging/jaahd@.service` gives the vnode daemon its own mount namespace: `BindPaths=` (lines 74 and 78), `ProtectHome=tmpfs` (line 80), `InaccessiblePaths=`, and `TemporaryFileSystem=` (lines 87 and 90). It sets no `MountFlags=`. The runner (`jaah-runner-vn@.service`) and GHA (`jaah-gha-vn@.service`) units are similar.
- The helper binds with plain `MS_BIND` (`apps/jaahd/vnodehelper/nodelinks.py` lines 49 and 160-164). Nothing in `apps/jaahd` sets mount propagation (no `MS_SHARED`, `MS_SLAVE` or `--make-*`). So a bind made AFTER the units start reaches them only if the host-side parent mount is shared and the unit's namespace is a slave of it.
- `apps/jaahd/vnodehelper/fence.py` lines 26-28 and the vnodehelper README (line 52) assert "slave propagation keeps the unit's view live". No test proves it.
- Ordering:
- `ops.start()` (`apps/jaahd/vnodehelper/ops.py` around lines 1355-1360) restores NodeLinks BEFORE the units start, so those mounts are visible by copy.
- The live converge (`ops.py` `_converge_shared` and `set_shared`, around lines 787-803, calling `nodelinks.converge`/`mount_one`) runs AFTER the units start. That is the path whose mounts are not seen.
- The vnode daemon's "mounted" test is `os.path.ismount` inside its own namespace (`apps/jaahd/shared_storage.py` `_is_mount` lines 67-71, `nodelink_mount_states` lines 212-246, `mounted_general_nodelinks` lines 267-299). So `not_mounted` really means "not a mount in the unit's namespace".
## Fix
1. Confirm the cause on a co-located node:
- Compare the vnode jaahd's `/proc/<pid>/mountinfo` with the host's after a live converge. Note the propagation fields: `shared:N` / `master:N`.
- If node-shell access is needed and is not available, ASK (`Access`).
2. Make post-start binds reach every vnode unit namespace (daemon, runner, runner-cli, gha). Pick the least-surface option that works and say why. Candidates:
- (a) Make the helper's bind targets propagate: keep a shared mount above the vnode's jaah-home on the host, and confirm the units' namespaces are slaves of it. Add a test of the assumption fence.py asserts.
- (b) Bind inside each unit's namespace (`nsenter -m`/`setns`), using the same O_NOFOLLOW fd walk.
- (c) Restart the vnode's units after a NodeLink set change. This is acceptable only if it never kills a live runner pane. If it would, reject this option.
3. Revocation (`nodelinks.revoke`, by mount identity) must still remove the mount from every unit namespace. A detached NodeLink must disappear inside the vnode too.
4. Check the wizard-born path: a vnode created with a NodeLink already ticked must report it `mounted` without a manual step.
Out of scope: per-NodeLink uids, the hermetic-environment opt-in, file wiping on delete, dropping the legacy accounts mount, any UI change (a separate follow-up, 5861), and the plain-node converge.
Done means:
- Unit tests in `apps/jaahd` (vnodehelper and/or `vnode_converge`) prove the chosen mechanism. The test must fail with the fix reverted.
- Live on app.dev.jaah.ai: a co-located node with a general NodeLink attached AFTER it started shows that NodeLink `Mounted` in Node detail › NodeLinks within one converge. Detaching it shows it gone. A task on that node prints `$JAAH_NODELINKS` with the path. Cite screenshots and the task id.
- A vnode born from the Add-node wizard with a general NodeLink ticked reports it `Mounted`.
- `./dc check full` is green, the PR is merged to develop, and the PR's Jaah Deploy succeeded. Cite the PR number, merge sha and deploy run. Confirm the host runs a jaah package version containing the fix (shown in Node detail).
Outcome
Fixed: vnode setgid home made the helper leave NodeLink dirs root-only 2700, so vnodes saw nothing. PR #5093 merged, deployed; dev vnodes show NodeLinks Mounted (post-start, wizard-born, detach). Left: $JAAH_NODELINKS task print (git creds; 5863).
Artifacts
Activity
- Ready
- Runner started
- Awaiting response
- Awaiting response
- Runner resumed
- Runner ended (done)