๐Ÿšจ An enabled node is running a release behind the published one

Task 6969 ยท Done ยท Adhoc ยท created ยท updated

The control plane's `node-liveness` job (`apps/api/app/node_liveness.py`) found a violated invariant: > **An `enabled` node must be running the published jaahd release.** These nodes are heartbeating โ€” this is NOT a liveness alarm โ€” but the release they report (`config_snapshot->>'version'`) is strictly OLDER than the desired release the control plane serves them (`GET /api/agent/release-desired` โ€” the version the control plane is ITSELF running, since CP and .deb ship from one commit), so they are running old code while the control plane counts them as current capacity. This is the failure class that ran undetected for 6.4 days: `aws1`/`aws2` heartbeated perfectly while frozen 796 commits back, because their self-update had failed every ~640s since it was armed (#1899). A stale node is not a dead node, and it is the more dangerous of the two mid-migration. - **h1** โ€” generation v1.2.1075 (behind), reported 2.2h ago - THERE IS NO PUSH TO RETRY โ€” every node self-converges, cloud and desktop by the identical path (ADR-0062, ADR-0107). The daemon asks this control plane what it should be running on each node-bus reconnect and drops `/run/skynet/requests/upgrade`; the root `jaahd-upgrade.path` unit sees the marker and runs `jaahd-update.service`, with `jaahd-update.timer` as a ~30min fallback poll (`apps/jaahd/upgrade_request.py`, `/usr/libexec/jaahd-update`). - so DIAGNOSE ON THE NODE ITSELF, over Tailscale SSH: `systemctl status jaahd-update.timer jaahd-upgrade.path jaahd-update.service`, then `journalctl -u jaahd-update.service -n 200` for the `jaahd-update:` lines (a node still on the pre-rename `skynet` package names all three `skynetd-*` and logs `skynetd-update:`) โ€” every pass that skips names its own reason (auto-update off for the node in Settings, no `/etc/skynet/skynetd.env`, the CP unreachable or answering an HTTP status, or the version not yet published to the apt repo). A `jaahd-upgrade.path` that is not enabled is the silent one: the node then converges only on the 30min timer, and nothing else anywhere says so. - the remedy is a DEPLOY, not a restart: a restart relaunches the same installed package. The desired version is this control plane's OWN version and has no writer to move, so a node that still will not catch up is an apt, credential or unit problem ON THAT NODE โ€” not a pointer to re-push at it. `apt-get install jaah=<version>` on `h1` is the manual last resort; dpkg's postinst restarts the daemon. - if it must not serve work until then: set `enabled = f` on its `nodes` row. Resolve this Task once every enabled node reports the published release. <!-- sysdedup:fleet-node-liveness-stale-release -->

Outcome

condition cleared โ€” fleet-node-liveness-stale-release

Activity

  • Draft
  • Runner ended (done)