Rootless Docker vnode hosting, efs-utils

## Original requirement (verbatim) > Make sure Jaah nodes can install and work on Graviton-based AWS instances, earlier version of Ubuntu. It is ok if some features might not work, such as microVM. Should work with the same installation command with any arch specifics inside the bootstrap Operator scope additions (verbatim, PRD task 11088): "Install higher version of python on older versions of Ubuntu." · "Also add to scope Amazon Linux two latest versions." (resolved: AL2023 + the newest after it = AL2027 preview, architecture O2) · "Both graviton and x64 must work on supported versions of ubuntu or amazon linux" · "24.04 is supposed to have rootless Docker. Double check the sources" **Supported matrix S** (single home: `packaging/linux-support.toml`, created by the Foundation child "Report Linux distro/arch, gate capabilities"): x86_64 + aarch64 × Ubuntu 22.04, 24.04, 26.04, Amazon Linux 2023, Amazon Linux 2027 (preview) = 10 cells. Design inputs (read with `task_read`): PRD = task 11088, architecture = task 11089. §/D/R references below are the architecture's; C-n are the PRD's acceptance criteria. D18 applies to every build child: prove pre-merge with unit/integration tests plus container tests (`docker run --platform linux/{amd64,arm64}` under qemu binfmt, mem-26702 method); post-merge, after Jaah Deploy to dev succeeds, launch the child's own EC2 cells and fix forward. ## Scope (architecture Child E; §3.1 rootless_docker, D11, D13, R3, R6, PRD C-6) 1. **Ubuntu 22.04/24.04**: prove or fix the docker.io + rootlesskit route (`infra/jaah/node_reprovision.py:125-134,940-958`; contrib `dockerd-rootless-setuptool.sh`, `rootlesskit`, `slirp4netns` where rootlesskit predates gVisor networking). Never assume either way: on a real 24.04 instance per arch, run it and record the exact command/output. 2. **Amazon Linux 2023/2027 (D13, O3)**: pin `docker-rootless-extras-<ver>.tgz` (download.docker.com static, x86_64 + aarch64) with `<ver>` = the engine version of AL's `docker` package; install `rootlesskit`, `dockerd-rootless.sh`, `dockerd-rootless-setuptool.sh` to `/usr/local/bin`; use rootlesskit's `pasta` network driver (AL ships `passt`; no `slirp4netns`; bundled `vpnkit` is x86_64-only). Via `infra/jaah/pkgmgr.py` (Amazon Linux child). 3. **Verified marker → `features.rootless_docker`**: convergence writes `/var/lib/jaahd/rootless-docker.ok` only after `docker info` as the runner user against its rootless daemon shows `rootless` in Security Options; any later failure removes it (R3: capability turns off rather than lies). Converged **from postinst's detached toolchain layer on every install and upgrade**, not only reprovision. The daemon's probe always emits `rootless_docker` (true iff the marker exists) from this release on. `hosts.lacks_engine` (`hosts.py:713-725`) stays the runtime backstop. 4. **efs-utils (D11)**: rebuild our `.deb` in `ubuntu:22.04` (`tools/build_efs_utils_deb.sh:27` is `ubuntu:26.04` today) so one binary per arch runs on 22.04/24.04/26.04. 5. Correct the stale comment at `apps/api/app/aws/ec2.py:1111` to the observed truth (the only allowed edit in that file). Credentials & reach: - Launch and terminate tagged EC2 test cells (post-merge) — `me/jaah/system/bootstrap-admin` → assume `OrganizationAccountAccessRole` in the `jaah-dev` org account, `us-east-1` — **verified** (broker fetch succeeded; `sts get-caller-identity` returned the assumed role in jaah-dev). O4 authorises mutations, limited to instances tagged `jaah:test=task-<id>`. - AMIs for every cell — SSM public parameters `/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-6.1-{x86_64,arm64}`, `…/al2027-preview-ami-kernel-7.1-{x86_64,arm64}`, `/aws/service/canonical/ubuntu/server/...` — **verified** (listed from h2). - Dev console UI and API, incl. C-7 API refusals — `org/console/dev-login` (operator) at `https://app.dev.jaah.ai` — **verified** (broker fetch succeeded; console returned 200 from h2). - Existing dev nodes for C-9 — dev console **Nodes** list, signed in as `org/console/dev-login` — **verified** (inventory at design time: 11 Ubuntu x64, 1 Ubuntu arm64, 1 Windows x64). - Deploy to dev (C-5, C-9) — merge → `jaah-deploy.yml` develop path — needs nothing beyond the worktree. - Container tests (D18) — docker + qemu binfmt on the runner's node (mem-26702) — needs nothing beyond the worktree host. - Gates — `./dc check quick` / `full` — needs nothing beyond the worktree. Out of scope: Do NOT touch / do NOT do — - Do NOT change `UBUNTU_RELEASE`/`_UBUNTU_IMAGE_NAME` or anything in `ec2.py` beyond the `:1111` comment. - Do NOT infer `rootless_docker` from tool presence (C-6). - Do NOT touch MicroVM. - Do NOT edit the capability rule table except if a real-machine run disproves vnode hosting on a cell (then the existing `vnode_host` row reports it via the false marker). - Epic-wide negative constraints (PRD §5, verbatim): - MUST NOT make MicroVM work on arm64 Linux, or on any newly supported release. It is reported unavailable (C-7) unless it already works there. - MUST NOT change the cloud-launched EC2 image (`apps/api/app/aws/ec2.py` `UBUNTU_RELEASE` / `_UBUNTU_IMAGE_NAME`) or add an OS picker to the EC2 create flow. The stale comment at `ec2.py:1111` is the one allowed edit there (C-6). - MUST NOT change the macOS or Windows installers, bundles or their install commands (`node_bootstrap.py` macOS and Windows sections, `packaging/macos/`, `packaging/windows/`). - MUST NOT replace, relink or upgrade the distro's system `python3` (`/usr/bin/python3`). - MUST NOT add an OS or arch argument, flag or variant to the console's install command. - MUST NOT support Ubuntu 20.04, Debian, RHEL, Amazon Linux 2, or architectures other than x86_64 and arm64 as **supported**. They get only C-8's warn-and-try. - MUST NOT gate the support behind a feature flag, plan feature or setting. - MUST NOT bump unrelated pins in `packaging/platform-pins.toml`, `packaging/microvm-pins.toml` or `packaging/runner-cli-locks/` beyond what a supported cell needs to install. Done means: - Unit tests for marker semantics (written only after a rootless `docker info`; removed on failure; probe emits the key true/false; postinst toolchain layer invokes convergence on install and upgrade). - Container oracle: rootless `docker info` where qemu allows it; efs-utils deb built on 22.04 runs `mount.efs --version` and `efs-proxy --version` on ubuntu:22.04/24.04/26.04 — cite output. - Post-merge real dev EC2: Ubuntu 24.04 × x64 and arm64 — `sudo -u jaah docker info` reports rootless and a vnode attached in the console comes up (C-6), or the exact failing command/output is recorded and the node reports `vnode_host` unavailable with the reason (C-7); AL2023/AL2027 vnode attach on real EC2 is proven by the sweep child (AL nodes enrol only through the bootstrap child's dnf branch, which this child does not wait on); here the AL path is proven by the container/unit oracle above. - NodeLink mounts on a 22.04 node and on an existing 26.04 node after deploy (R6); an existing 26.04 vnode host stays available (R8). - `ec2.py:1111` comment states the observed truth. - `./dc check quick` green; merge lane `./dc check full` green; Jaah Deploy to dev succeeded. PRD acceptance criteria this child carries (verbatim; prove the cells named above, others are proven by their owning child or the sweep): | ID | Criterion | Verify (oracle) | |---|---|---| | C-6 Rootless Docker / vnode on 24.04 | On Ubuntu 24.04 (both arches), rootless Docker for the runner user works, and the node can host a vnode. If a real-machine run proves it impossible, the build reports the exact failing command and output and treats vnode hosting as unavailable on that cell under C-7; it is never assumed either way. The stale comment at `apps/api/app/aws/ec2.py:1111` is corrected to the observed truth. | On the 24.04 instances: `sudo -u jaah docker info` succeeds and reports rootless. In the console, attach a vnode to the node and observe it come up. | | C-7 Unavailable features are reported and hidden | Some features cannot work on a cell, e.g. MicroVM on any arm64 Linux, vnode hosting where C-6 disproves it, EFS NodeLink where no helper package exists for that OS. Each one is reported by the node as unavailable with a one-line human reason. The console does not offer that feature for that node and shows the reason where the feature would appear. The install still succeeds (C-2). This holds for every capability, on every console surface that offers one: node detail, MicroVM toggle, vnode attach, NodeLink attach, runner-provider choice. | For each cell: in dev.jaah.ai open the node's detail and every place that offers a node capability. Each unavailable feature is absent or disabled, with a readable reason (not truncated, not a slug). Attempting it via the API returns a refusal naming the reason, never a 500 or a hang. Each available feature is offered. |

Merged #7309 (20d7074e7, lane full gate green): verified rootless Docker (pinned docker-rootless-extras) on Ubuntu 22.04/24.04 + AL2023/2027, rootless_docker marker+probe, efs-utils deb built on 22.04 (runs 22.04–26.04). Container oracle rootless on all x64 cells; arm64 qemu partial. Deploy superseded; real-EC2 C-6/R6/R8 cells not run (deferred to sweep). Memory: none applied.