๐Ÿšจ An enabled node connects but is not observing

Task 11327 ยท Done ยท Adhoc ยท created ยท updated

The control plane's `node-liveness` job (`apps/api/app/node_liveness.py`) found a violated invariant: > **An `enabled` node that reaches the bus must also be OBSERVING.** These nodes are on the bus (`last_seen` is fresh) but their observation passes are not landing, so they contribute nothing while the control plane counts them. A different condition from silence, with a different remedy: the heartbeat path is healthy and the observation/ingest path is not. The bound is `10 x 60s = 600s`: 10 missed reports at the 60s cadence `jaah_discovery._LAST_SEEN_REFRESH_INTERVAL` debounces the `last_seen` write to โ€” the fastest the column this bound is measured over can advance. - **eduardo-local-machine** โ€” last observation 12.7h ago (2026-10-29 04:40:52+00), but last seen 393s ago (2026-10-29 17:13:31.062958+00) - complete slot enumeration (`worktrees_updated_at`, the clock `apps/api/app/pool_state.py` gates DISPATCH on โ€” evidence, not this alarm's subject): 421s ago - if it SHOULD be observing: check the node's observation POST path (its `jaahd` is running, since the heartbeat arrives โ€” the failure is downstream of presence). - if it should NOT: set `enabled = f` on its `nodes` row. - **eduardo-local-machine-host** โ€” last observation 12.7h ago (2026-10-29 04:40:52+00), but last seen 393s ago (2026-10-29 17:13:31.691162+00) - complete slot enumeration (`worktrees_updated_at`, the clock `apps/api/app/pool_state.py` gates DISPATCH on โ€” evidence, not this alarm's subject): 421s ago - if it SHOULD be observing: check the node's observation POST path (its `jaahd` is running, since the heartbeat arrives โ€” the failure is downstream of presence). - if it should NOT: set `enabled = f` on its `nodes` row. Resolve this Task once every enabled, connected node is observing inside the bound. <!-- sysdedup:fleet-node-liveness-not-observing -->

Outcome

condition cleared โ€” fleet-node-liveness-not-observing

Activity

  • Draft
  • Runner ended (done)