๐จ An enabled node connects but is not observing
The control plane's `node-liveness` job (`apps/api/app/node_liveness.py`) found a violated invariant:
> **An `enabled` node that reaches the bus must also be OBSERVING.**
These nodes are on the bus (`last_seen` is fresh) but their observation passes are not landing, so they contribute nothing while the control plane counts them. A different condition from silence, with a different remedy: the heartbeat path is healthy and the observation/ingest path is not.
The bound is `10 x 60s = 600s`: 10 missed reports at the 60s cadence `jaah_discovery._LAST_SEEN_REFRESH_INTERVAL` debounces the `last_seen` write to โ the fastest the column this bound is measured over can advance.
- **eduardo-local-machine** โ last observation 12.7h ago (2026-10-29 04:40:52+00), but last seen 393s ago (2026-10-29 17:13:31.062958+00)
- complete slot enumeration (`worktrees_updated_at`, the clock `apps/api/app/pool_state.py` gates DISPATCH on โ evidence, not this alarm's subject): 421s ago
- if it SHOULD be observing: check the node's observation POST path (its `jaahd` is running, since the heartbeat arrives โ the failure is downstream of presence).
- if it should NOT: set `enabled = f` on its `nodes` row.
- **eduardo-local-machine-host** โ last observation 12.7h ago (2026-10-29 04:40:52+00), but last seen 393s ago (2026-10-29 17:13:31.691162+00)
- complete slot enumeration (`worktrees_updated_at`, the clock `apps/api/app/pool_state.py` gates DISPATCH on โ evidence, not this alarm's subject): 421s ago
- if it SHOULD be observing: check the node's observation POST path (its `jaahd` is running, since the heartbeat arrives โ the failure is downstream of presence).
- if it should NOT: set `enabled = f` on its `nodes` row.
Resolve this Task once every enabled, connected node is observing inside the bound.
<!-- sysdedup:fleet-node-liveness-not-observing -->
Outcome
condition cleared โ fleet-node-liveness-not-observing
Activity
- Draft
- Runner ended (done)