GHA orphan runner users never swept

Task 5792 · Done · Coder · created · updated

## Defect Epic 5758 requires "Down-sizing a vnode's runner pool leaves no orphan gha home; aws1-2's `gha2..gha4` are gone" (AC-5.2). C3 (PR #5013) added the orphan sweep, but on prod it has not fired. ## Reproduction (observed by the finish runner, 2026-10-11 ~18:30 UTC, prod console, operator login) - aws1-2 = vnode `srcdyny6` (Properties › Jaah home `/var/lib/jaah-vn/srcdyny6/home/.skynet`). Runners tab: pool 2/2, gha-1 and gha-2. Node on 1.1.8978. - aws1-2 › Health › Disk, the 18:21 UTC scheduled reclaim on host aws1-host. Its `detail.engines` still lists rootless engines for `vnsrcdyny6-gha2` (0 B), `vnsrcdyny6-gha3` (0 B) and `vnsrcdyny6-gha4` (4.37 GB). The canonical users exist above the count, and their user managers and dockerd are live. No converge event has had `removed_users` non-empty since the release (~17:14 UTC). ## Cause (read from code; host state not verified) The sweep runs only inside a gha converge (`apps/jaahd/vnodehelper/gha.py:334` → `remove_surplus`). A converge is started only by a marker: - `gha_request.request_converge`, `apps/jaahd/gha_request.py:123`. - It is written by a console Apply (`apps/jaahd/gha_runner_ops.py:436-444`) or by the 60 s sensing pass when `converged` is False (`gha_runner_ops.py:375-399`). `converged = matches_desired(...) and not outstanding_removals` (`gha_runner_ops.py:376`, `:201-248`) looks only at observed runner UNITS. An orphan user with no unit never makes the pool look unconverged, so a steady pool never writes a marker and never sweeps. A second possible blocker: `install.py::orphan_refusal` (`infra/host/gha-runners/install.py:1883-1908`) keeps an account forever when a leftover unit (`slot_units`) or runner tree is present. Check that too. ## Fix - In the sensing pass, treat "a canonical gha user (`layout.gha_user` for a vnode, `install.slot_account` for a host) exists at an index ≥ the desired count and is not busy" as NOT converged. The sensing pass then asks for one converge, which runs the existing sweep. Reuse `vnodehelper.gha.orphan_indices` and its host twin; do not add a second scanner. - The existing repeat gate (`gha_request.py:58-61`) must still bound re-asks. A user that `orphan_refusal` keeps must not cause a marker every 150 s forever. Key the ask on the orphan set, or log the refusal and stop asking for an unchanged set. - If `orphan_refusal` keeps aws1-2's users only because of a leftover unit or tree from an older install that has no live runner, extend the sweep to remove that dead unit or tree first. Use the same canonical-name rule, and never touch a busy slot. ## Out of scope - Do not change the ephemeral host-slot model or `gha_job.py` (D8). - Do not change the canonical-name rule or match users by regex. - Do not change reclaim (`disk_reclaim.py`), the CP gate or the console. - Never remove a busy slot or kill a job. - No SSM or host-root access. Prove it through the console. ## Done means - Unit test: a vnode and a host pool with desired count M, unit-less canonical users at indices ≥ M and nothing busy. The sensing pass reports not converged exactly once and writes one marker. After the sweep it reports converged and writes no further marker. A refused orphan (unit present) does not re-ask every cycle. - Unit test: a busy orphan index is never swept. - Live, through the prod console (operator login), after release: a converge event on aws1-2 lists `vnsrcdyny6-gha2`, `-gha3` and `-gha4` in `removed_users`. Later reclaim events' `detail.engines` no longer list those three users. - The `infra/host/gha-runners/README.md` orphan paragraph states what triggers the sweep. `./dc check full` green; PR merged; Jaah Deploy succeeded.

Outcome

Sensing pass now asks for a converge on unit-less orphan gha users. Gate green, PR #5044 merged, dev and prod deploys succeeded. aws1-2 gha2-4 engines are gone from the 20:07 UTC reclaim. Converge event naming them not yet seen; event upload is lagging.

Pull request

Artifacts

Activity

  • Ready
  • Runner started
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)