MicroVM end-to-end acceptance on dev

Epic 5777 — MicroVM nodes. This child is the architecture's **C12: End-to-end acceptance on dev**. It runs after every build child (phase 2). Through the **dev console only** — no host shell; "nothing left" is read from the node page's MicroVM panel (`vms`/`vm_dirs`/`ch_pids`/`virtiofsd_pids`/`taps_busy`). Any failure → fix it if it is small and in-scope (feature branch + PR, per the project's delivery rules), otherwise file a laser-focused follow-up and name it. Upstream: PRD task 5778 `prd`, Architecture task 5779 `architecture`. ## Original requirement (verbatim) Epic Done means (as amended by the operator interview, I4/I5): - "A real task in a `dynamic` project completes end to end (dispatch → gate green → auto-merged PR → `done` → VM destroyed, nothing left) on BOTH a nested-virtualization m8i cloud node AND a physical KVM host, using a NodeLink subscription account; the refreshed token is observed from a second node." ("dynamic project" = project assigned to a MicroVM node.) - "From inside a running VM, each of these is demonstrated to FAIL: reading a sibling task's worktree or token dir, listing other accounts, connecting to port 2049, connecting to 169.254.169.254, reaching a domain not on the allowlist." - "Console attach to the running task and delivery of an answer to a waiting question are demonstrated." - "ADR-0129 amended; hosting products updated; ... a node can be marked MicroVM from the console" (amended). - Measured comparison (cold start, gate wall time, memory) is **deferred** by the operator (I4: "Out of scope. We just build regardless."). A11: "The NodeLink end-to-end run (AC-1) uses a Claude subscription account. Codex and Gemini are each shown by one real task run in a VM on the m8i node, with either an API-key or a NodeLink account." I12: "I will create one in GMind or Timesaver organization" (the physical KVM host). ## Acceptance criteria (verbatim — execute each) - AC-1 End to end, two node kinds, NodeLink. Oracle: on dev, mark an m8i co-located node (nested virt) MicroVM in the console. Assign a test project. File a real coder task, run with a Claude NodeLink subscription account. Observe dispatch → gate green inside the VM → PR auto-merged → task `done` → VM gone: no cloud-hypervisor process, no VM disk, no tap/vsock socket for it on the node (`ps`, `ls` of the VM state dir). Repeat on the operator-created physical KVM host in GMind or Timesaver (I12). On the physical host the runner ASKS the operator for the host when it gets there. **Fails if** either run needs any manual step on the node, or leaves any VM artefact behind. - AC-2 Token write-back. Oracle: during AC-1, force a token refresh inside the guest. On a second node mounting the same org store, read the account's `.credentials.json` modification time and expiry. Observe that they match the refresh. **Fails if** the second node still shows the pre-refresh token. - AC-3 Every provider. Oracle: run one real task each with Codex and with Gemini in a VM on the m8i node, through to `done`. **Fails if** any provider is refused or needs an API-key-only path that the others don't. - AC-4 Negative isolation (all five must FAIL from inside a running VM). Oracle, from the guest's shell: `ls` / `cat` a sibling task's worktree path and token dir (absent or permission denied); list the NodeLink accounts dir (absent); `nc -zv -w3 <efs-ip> 2049` (fails); `curl -m3 http://169.254.169.254/` (fails); `curl -m5 https://example.org` with example.org not on the allowlist (refused by the proxy). Then add example.org to the project's allowlist in the console, start a new task, and show the same curl succeeds. **Fails if** any probe succeeds, or the allowlisted one does not. - AC-5 Identity cannot be forged. Oracle: from guest A, call a control tool for its own task and see it succeed. Craft a request that claims task B (forged id field, fake uid, or a process tree that mimics B) and see it refused, and nothing written to B. **Fails if** anything in the guest's request changes the resolved task. - AC-6 Control calls unchanged. Oracle: inside the guest, `jaah-forge pr view`, a `git push` of the task branch (uses the credential helper), `jaah-call task.get` and a `task_artifact` upload all succeed, with no command changed versus a static runner. - AC-7 Docker in the guest. Oracle: in two concurrent VMs on one node, `./dc up` the project's stack on default ports. Both are healthy, with no port-grid offset. - AC-8 Console attach and answer delivery. Oracle: open the running VM task in the dev console terminal and see live output; type a key and see it echo. Make the task ask an `AskUserQuestion`, answer in the console, and see the session continue with the answer. Repeat attach for a human console session and an advisor chat on a MicroVM node, each shown to be in its own VM (`systemd-detect-virt` inside reports a VM, and a distinct VM id). **Fails if** the terminal is read-only, lags without updating, or the answer does not arrive. - AC-9 Guards hold. Oracle: from the guest, `git push origin HEAD:develop` is refused. A settle with an uncommitted file is refused. The Stop gate runs. - AC-10 Sizing and sharing. Oracle: set the System, org, project and node values in the console to distinct sizes. Launch a VM and read inside it (`nproc`, `free`, `df`). It has the **node** value, and with the node value cleared it has the project's. Run more VMs than the sum of RAM caps would allow. They are admitted while real free memory clears the reserve, and the next is refused with a readable reason once it does not. A guest's disk file on the host is smaller than its cap (thin). - AC-11 Marking a node, available to the operator (R29). Oracle: in the dev console, as an org admin on a plan with the feature, open a KVM node and turn MicroVM on. It saves and shows. On a node without KVM, the control is refused, with a sentence saying KVM is missing. On a plan without the feature, it is disabled with a reason (flip a test plan's toggle in the dev console and restore it). On a shared node, as a tenant admin, it is not offered, and as the operator it works (create the shared node through the dev console as operator if none exists, Q10, and remove it after). On a node holding a leased static slot, turning it on is refused with the blocker sentence (A1). The feature is **ON by default** for every plan the operator configures with it, and no hidden flag stays off in dev or prod. **Fails if** an operator cannot do any of this from the console alone. - AC-12 (UI step): the dev project settings page shows no slot-mode control. - AC-13 Launches enable nested virt. Oracle: launch a dev cloud node on m8i through the console. `aws ec2 describe-instances` (read-only) shows nested virtualization enabled. The node reports `kvm = true` in the Hosts page. - AC-15 (live): a fresh node downloads the Ubuntu cloud image and verifies its digest (observed on the node page's base state). - AC-16 Gate. Oracle: `./dc check full` is green on the delivering PRs, and the Jaah Deploy for dev succeeds. - Also record the deferred runtime proofs from the build children: K1 (non-root tap), K2 (console socket mode), K8 (balloon pages return to `MemAvailable`; if not, ask the operator per the architecture: virtio-mem hotplug vs accept start-up-need admission), K3 (one `./dc check quick` wall time in a VM, evidence only). - Prod-side items that need a human-gated `master` promotion (prod catalogue "MicroVM" column, AC-11 "no hidden flag off in prod") are reported as "verified after promotion" — a named deferred follow-up, not claimed. - Project instruction: while testing via the UI, note anything broken, inconvenient, not user friendly or suboptimal and file a task with itemized 1–2 sentence suggestions. Credentials & reach (Architect R31, verbatim status): - AC-1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 UI steps (dev console) — `jaah/debug/main/operator-login` (operator + tenant switch) on `https://app.dev.jaah.ai` — **deferred**: credential verified, but reach is refused from fleet nodes (admin prefix list). The operator answered (Q7) they will add fleet egress to dev's admin prefix list. Re-probe `curl -s -o /dev/null -w %{http_code} https://app.dev.jaah.ai` at start; if still refused, ask the operator once (header `External`), and otherwise deliver what is reachable and settle with the remainder named. - AC-13 read-back (describe-instances shows nested virt on) — `jaah/system/bootstrap-admin`, assuming `OrganizationAccountAccessRole` in org account `jaah-dev`, region us-east-1 — **verified** (role assumption and read-only `ec2 describe-instances` succeeded). - AC-13 launch step, AC-1 m8i co-located node launch — dev console (operator login), using the dev org's own AWS credential the console launch already holds — **deferred** (dev reach, Q7). - AC-15 fresh-node image download — a dev MicroVM node observed on its node page (`microvm.base.state`) — **deferred** (dev reach, Q7). - AC-3 Codex and Gemini — the existing dev Codex and Gemini runner accounts (Q9) — **deferred** for identification only: find them on the dev console's runner accounts page; ask only if absent. - AC-11 shared-node check — a shared node on a dev operator host, created through the dev console as operator (Q10), tenant-admin view via the operator login's tenant switch — **deferred** (created at run time, gated on dev reach). - AC-1/AC-4 "nothing left" and in-guest probes — no host shell: the dev node page's `microvm` block and console attach to the guest shell, as the operator login — **deferred** (dev reach, Q7). - AC-1 physical KVM host — operator-created host in GMind or Timesaver org (I12) — **deferred**: ask the operator for the host when you reach that step. - AC-2 NodeLink account + second node — existing Claude NodeLink subscription account in a dev org and a second node on the same store (Q8) — **deferred** for identification only: find it on the dev console's NodeLink accounts page; ask only if none. - AC-16 — forge login via `jaah-forge` (broker) — **verified**. Out of scope: no measured comparison/benchmark (N9, I4). Do not migrate or relaunch existing t4g hostings (N6). Do not edit product data by SQL/CLI/API (N8). Do not operate Jaah state through any CLI/SQL/raw API — dev console only (hub rule); the AC-13 read-only `describe-instances` is the one settled exception. Do not build new features here — only fixes for failures found, each as its own small PR, or named follow-ups. Epic: "retiring static worktrees, co-located nodes (Multi-Node Hosts), the port grid or user slots; Mac/Windows hosts; GPU passthrough; any change to pricing beyond moving products to KVM-capable instance families." Done means: - Each of AC-1 … AC-13, AC-15, AC-16 above executed with its observed result cited in the report (screenshots/console output); any `deferred` item reported with its recorded reason, not silently skipped. - AC-1 on the m8i co-located node: node page shows `vms` empty, `vm_dirs` empty, `ch_pids` 0, `virtiofsd_pids` 0, `taps_busy` 0 after `done`. - AC-1 on the physical KVM host (operator asked for it): same. - AC-4: all five probes fail; allowlist edit makes example.org succeed. - K1/K2/K3/K8 outcomes recorded. - Every failure has either a merged fix PR (Jaah Deploy dev green) or a named follow-up task. - Prod-only checks named as post-promotion follow-ups.

Dev MicroVM C12: fixed 5 live blockers (#5127, #5139, #5150; dev deploys green). A task VM now boots on m8i vn5862b and tears down clean. AC-11/12/13/15 pass. Deferred: AC-8 VM attach (F1), repo PR path (no dev GitHub launcher), physical host, in-guest probes.