๐Ÿšจ An enabled node connects but is not observing

Task 11973 ยท Done ยท Adhoc ยท created ยท updated

The control plane's `node-liveness` job (`apps/api/app/node_liveness.py`) found a violated invariant: > **An `enabled` node that reaches the bus must also be OBSERVING.** These nodes are on the bus (`last_seen` is fresh) but their observation passes are not landing, so they contribute nothing while the control plane counts them. A different condition from silence, with a different remedy: the heartbeat path is healthy and the observation/ingest path is not. The bound is `10 x 60s = 600s`: 10 missed reports at the 60s cadence `jaah_discovery._LAST_SEEN_REFRESH_INTERVAL` debounces the `last_seen` write to โ€” the fastest the column this bound is measured over can advance. - **HomeJab** โ€” last observation 918s ago (2026-10-31 18:54:52+00), but last seen 310s ago (2026-10-31 19:04:59.486867+00) - complete slot enumeration (`worktrees_updated_at`, the clock `apps/api/app/pool_state.py` gates DISPATCH on โ€” evidence, not this alarm's subject): 913s ago - if it SHOULD be observing: check the node's observation POST path (its `jaahd` is running, since the heartbeat arrives โ€” the failure is downstream of presence). - if it should NOT: set `enabled = f` on its `nodes` row. - **HomeJab-host** โ€” last observation 913s ago (2026-10-31 18:54:57+00), but last seen 310s ago (2026-10-31 19:04:59.502238+00) - complete slot enumeration (`worktrees_updated_at`, the clock `apps/api/app/pool_state.py` gates DISPATCH on โ€” evidence, not this alarm's subject): 912s ago - if it SHOULD be observing: check the node's observation POST path (its `jaahd` is running, since the heartbeat arrives โ€” the failure is downstream of presence). - if it should NOT: set `enabled = f` on its `nodes` row. Resolve this Task once every enabled, connected node is observing inside the bound. <!-- sysdedup:fleet-node-liveness-not-observing -->

Outcome

condition cleared โ€” fleet-node-liveness-not-observing

Activity

  • Draft
  • Runner ended (done)