GHA orphan runner users never swept
## Defect
Epic 5758 requires "Down-sizing a vnode's runner pool leaves no orphan gha home; aws1-2's `gha2..gha4` are gone" (AC-5.2). C3 (PR #5013) added the orphan sweep, but on prod it has not fired.
## Reproduction (observed by the finish runner, 2026-10-11 ~18:30 UTC, prod console, operator login)
- aws1-2 = vnode `srcdyny6` (Properties › Jaah home `/var/lib/jaah-vn/srcdyny6/home/.skynet`). Runners tab: pool 2/2, gha-1 and gha-2. Node on 1.1.8978.
- aws1-2 › Health › Disk, the 18:21 UTC scheduled reclaim on host aws1-host. Its `detail.engines` still lists rootless engines for `vnsrcdyny6-gha2` (0 B), `vnsrcdyny6-gha3` (0 B) and `vnsrcdyny6-gha4` (4.37 GB). The canonical users exist above the count, and their user managers and dockerd are live. No converge event has had `removed_users` non-empty since the release (~17:14 UTC).
## Cause (read from code; host state not verified)
The sweep runs only inside a gha converge (`apps/jaahd/vnodehelper/gha.py:334` → `remove_surplus`). A converge is started only by a marker:
- `gha_request.request_converge`, `apps/jaahd/gha_request.py:123`.
- It is written by a console Apply (`apps/jaahd/gha_runner_ops.py:436-444`) or by the 60 s sensing pass when `converged` is False (`gha_runner_ops.py:375-399`).
`converged = matches_desired(...) and not outstanding_removals` (`gha_runner_ops.py:376`, `:201-248`) looks only at observed runner UNITS. An orphan user with no unit never makes the pool look unconverged, so a steady pool never writes a marker and never sweeps.
A second possible blocker: `install.py::orphan_refusal` (`infra/host/gha-runners/install.py:1883-1908`) keeps an account forever when a leftover unit (`slot_units`) or runner tree is present. Check that too.
## Fix
- In the sensing pass, treat "a canonical gha user (`layout.gha_user` for a vnode, `install.slot_account` for a host) exists at an index ≥ the desired count and is not busy" as NOT converged. The sensing pass then asks for one converge, which runs the existing sweep. Reuse `vnodehelper.gha.orphan_indices` and its host twin; do not add a second scanner.
- The existing repeat gate (`gha_request.py:58-61`) must still bound re-asks. A user that `orphan_refusal` keeps must not cause a marker every 150 s forever. Key the ask on the orphan set, or log the refusal and stop asking for an unchanged set.
- If `orphan_refusal` keeps aws1-2's users only because of a leftover unit or tree from an older install that has no live runner, extend the sweep to remove that dead unit or tree first. Use the same canonical-name rule, and never touch a busy slot.
## Out of scope
- Do not change the ephemeral host-slot model or `gha_job.py` (D8).
- Do not change the canonical-name rule or match users by regex.
- Do not change reclaim (`disk_reclaim.py`), the CP gate or the console.
- Never remove a busy slot or kill a job.
- No SSM or host-root access. Prove it through the console.
## Done means
- Unit test: a vnode and a host pool with desired count M, unit-less canonical users at indices ≥ M and nothing busy. The sensing pass reports not converged exactly once and writes one marker. After the sweep it reports converged and writes no further marker. A refused orphan (unit present) does not re-ask every cycle.
- Unit test: a busy orphan index is never swept.
- Live, through the prod console (operator login), after release: a converge event on aws1-2 lists `vnsrcdyny6-gha2`, `-gha3` and `-gha4` in `removed_users`. Later reclaim events' `detail.engines` no longer list those three users.
- The `infra/host/gha-runners/README.md` orphan paragraph states what triggers the sweep. `./dc check full` green; PR merged; Jaah Deploy succeeded.
Outcome
Sensing pass now asks for a converge on unit-less orphan gha users. Gate green, PR #5044 merged, dev and prod deploys succeeded. aws1-2 gha2-4 engines are gone from the 20:07 UTC reclaim. Converge event naming them not yet seen; event upload is lagging.
Artifacts
Activity
- Ready
- Runner started
- Awaiting response
- Awaiting response
- Runner resumed
- Runner ended (done)