Don't place a new host's first node

Task 5055 · Done · Coder · created · updated

Defect (epic 4929, dev finding F6): on a freshly launched rented/cloud host the first node is placed before the host's rootless Docker layer exists (~8 min of first boot). The first vnode create fails `docker_failed`, the order shows failed after the tenant has paid, then recovers by itself when the host retries (F3 recovery). Observed on dev for beta and epsilon-2 (Task 4973 report). Cause: - `apps/api/app/node_orders.py:459` — `on_host_polled` gates placement only on `vnodes.host_not_up(host_row)`. - `apps/api/app/vnodes.py:482-495` — `host_not_up` checks `instance_id` (cloud) / `last_seen` (self-hosted), not engine readiness. - `apps/jaahd/vnodehelper/ops.py:866` — `ensure_rootless_engine` raises `OperationFailed("docker_failed")`. - Test pinning today's behaviour: `apps/api/tests/test_node_orders.py:585-594`. Fix (pick the smallest): either (a) treat `docker_failed` on a host's FIRST vnode while the host is younger than its first-boot window as a transient — the order stays at its "Joining" stage, never "failed" — or (b) have the host report an engine-ready fact the CP already ingests and gate `on_host_polled` on it. Prefer (a) if it needs no node-side change. Explain the choice in the PR body. Out of scope: no change to the vnodehelper isolation (nft, quotas); no new order stages in the UI beyond reusing existing labels; F3 recovery path stays as is. Done means: - New host, first vnode reports `docker_failed` within the first-boot window → order is NOT failed and the UI shows an in-progress stage; it reaches ready when the vnode reports ready — regression test replacing/adjusting test_node_orders.py:585-594. - A `docker_failed` outside that window (or on a non-first vnode) still fails the order with its sentence and Retry — test. - `./dc check full` green.

Outcome

A new host's first node failing docker_failed within a 20-min first-boot window keeps its order Joining; the sweep fails it past the window. Full gate green; PR #4489 merged. No UI change.

Pull request

Artifacts

Activity

  • Ready
  • Runner started
  • Runner ended (done)