Run MicroVM backend in WSL2 on Windows

## Epic requirement (verbatim) "Implement microVM on Windows and MacOS" (Task 10021 PRD ask #1, `in_scope`). Operator decisions that bind every child (verbatim from the PRD/Architecture): - D1 **Full parity** with Linux. Each session gets its own Linux guest kernel. The CP egress allowlist is enforced on the node. Control calls are vsock-bound (or an equivalent per-VM channel). No NodeLink inside the VM. No root in the guest. The disk is retained on `system_failed`. Console, sizing and placement UX are the same. - D2 **All three:** fleet runners, advisor chats and human console sessions. D3 **The Linux guest's set** of providers. D4 macOS = Apple Silicon, Windows = x64. D5 cloud Windows too. D7 same plan feature (`billing_plans.microvm_allowed` + `orgs.microvm_allowed_override`). - AD1 Windows = WSL2 + the existing Cloud Hypervisor/KVM runtime (Windows 10 refused). AD2 Mac guests arm64 only, no Rosetta. AD3 Windows test PC = `win-t8703`. AD5 probe m8i.large exactly; metal (m7i.metal-24xl) is the fallback. AD6 Mac oracles deferred until the Mac is enrolled on dev. AD7 AC2(a) live is dropped; unit tests map every `microvm_prereq` code to its sentence. **Quality bar (verbatim).** Every refusal shown to an operator is one plain sentence naming what is missing and how to fix it (`docs/standards/ui-copy-standards.md`). No generic 400, no stack trace, no Linux-specific wording on a Mac or Windows node. Boot-to-runner latency on Mac and Win is reported in the build's evidence. It is not gated, but it must not be degraded by more than 2× against a Linux MicroVM node of similar size without the Architect recording why. ## This child: C2 — Windows backend (Architecture §2 entire, §5 probe, §9.2) Requirements this child serves (PRD §4, verbatim): - R1. Mark. An Apple Silicon macOS node and an x64 Windows node (self-hosted, or cloud-launched for Windows) can be marked MicroVM through the existing Node detail toggle. Conditions: the org's plan allows MicroVM, the node reports a usable hypervisor, and it has no held static worktrees. When the node does not qualify, the refusal is a readable sentence naming the missing prerequisite. The Linux-only `NO_KVM` sentence must not be shown to a Mac or Windows node for a reason that no longer applies. - R2. Sessions in VMs. On a marked Mac or Windows node, every runner, advisor and console session runs in its own VM with its own Linux guest kernel. Each session is a `microvms` row, never a `worktrees` row. - R3. Parity of contract. Every bullet of `docs/policies/microvm-nodes.md` holds on these OSes: sizing resolver and source display; placement admission by free RAM/disk and hypervisor usability, with readable wait reasons, and `failing_nodes`; the intent/fact rail; lifecycle: destroy at done, reset, delete or console end; retain on `system_failed` for `microvm_retention_hours`; liveness reclaim; the egress allowlist: default ∪ project additions − removals, plus the CP and forge hosts, with private, link-local and node-own addresses refused and each refusal logged; per-VM control-call binding: a guest can act only on its own task; no root and no `sudo` in the guest, with a rootless engine; no NodeLink mount in a VM; the API-key sign-in served without a key in the guest environment. - R4. Providers. Inside a VM, a Mac or Windows node runs every provider an Ubuntu MicroVM node runs. Placement and the account picker must not refuse such a provider because of the host OS when the session lands in a VM. The host-session rules stay as the ADR-0145 table states for unmarked nodes. - R5. Cloud Windows. The AWS door can launch a Windows node on an instance type able to run MicroVM, and that node can be marked and run tasks. An incapable type gets a readable refusal, at launch when MicroVM is requested, or at the mark. - R6. Install and update. The macOS installer and the Windows service install and converge everything the VM runtime needs: binaries, images and helpers. The A2 prerequisite is the only manual step. Updating a node keeps its VMs' contract. - R7. Linux unchanged. Linux MicroVM behaviour, the static pool and the unmarked Mac/Windows behaviour do not regress. - R8. Docs describe the present. `docs/policies/microvm-nodes.md`, ADR-0129 (including the per-OS boundary statement), ADR-0145 / `node_platform.py` docstring, `apps/jaahd/README.md` (MicroVM section) and the macOS/Windows packaging docs describe the new state. Facts are single-homed, and `python3 tools/check_context.py` passes. The full settled Architecture (Task 10022) and PRD (Task 10021) are the epic's prerequisite artifacts; read them with `task_read`/`artifact_read` — §2 and §9.2 in full — before building. Section numbers (§) refer to the Architecture. ### Approach (settled, realize it) **First step — the probes (P2/P3), on dev through the console as the operator:** (1) on `win-t8703` read the reported `kvm`/`microvm_prereq`; (2) allow-list m8i.large, launch a Windows node on it through the AWS door, mark it MicroVM, observe `kvm`/`microvm_prereq`, then boot one VM at node size 1 vCPU / 2 GiB set on Node detail. The outcome sets `WINDOWS_NESTED_MICROVM` in `apps/api/app/instance_types.py`: boots → True, AC13 names **m8i.large**, record boot latency vs the 2× bar and why if exceeded; fails → stays False, AC13 names **m7i.metal-24xl** (allow-list it in the console), and the m8i node shows the readable `wsl_no_kvm` sentence (refusal half). Also record whether Server 2025 has WSL KVM; if not, change the server arm of `windows_build` to "no server build" and its sentence to "MicroVM on Windows needs Windows 11." (in the CP's `PREREQ_SENTENCES`). **Runtime (the Linux code, unmodified, in a second WSL2 distro):** - Distro `jaah-microvm` (Ubuntu 24.04, systemd on), imported at the **first mark-on** (rootfs pinned in `platform-pins.toml`, `platform = "windows-x64"`, id `ubuntu-noble-wsl-rootfs`; payload ships in the Windows bundle; not imported at install). `wsl.conf`: `[interop] enabled=false`, `[automount] enabled=false`. Runs `jaah-microvmd.service`, `jaah-microvm-converge.service` (root, taps + nft, inside the distro only), and `jaah-microvm-backend.service` (user `jaahd`, group `kvm`) hosting `microvm_rail`, `microvm_images`, `microvm_egress`, `microvm_session`, `microvm_liveness`, `microvm_facts`, `cred_broker` `VmCaller` — no logic change. CH/CLOUDHV.fd/virtiofsd: the same x86_64 pins (selected as `(x86_64, windows)` by the arm64-pins child's `tools/microvm_pins.py`). - **Bridge** `apps/jaahd/microvm_bridge.py` (both ends; the one home of the frame vocabulary and `BACKEND_CP_ROUTES`), pinned in §9.2: client process `wsl.exe -d jaah-microvm -u jaahd --exec /usr/lib/jaahd/bin/jaah-microvm-backend bridge` started by Windows jaahd only while marked, connecting to `/run/jaahd/backend.sock` (0600, user `jaahd`); frames `u32` BE length + UTF-8 JSON, max 1 MiB (larger closes; client reconnects); ops `hello` (`proto: 1`, payload = jaahd version; mismatch → Windows side converges `/usr/lib/jaahd/versions/<v>`, flips `current` with `wsl -u root`, `systemctl restart jaah-microvm-backend`, never microvmd), `mark`, `rail`, `cp` (+`cp_data`/`cp_end`/`cp_head`, 64 KiB chunks, 256 KiB credit window per `rid`/`sid` with `ack`), `host` (`{addrs, free_disk_mib}` every pass), `facts`, `attach`/`data`/`close`, `probe`; errors `unknown_op`|`bad_frame`|`forbidden_route`|`cp_unreachable`|`no_vm`|`cp_too_large`. Windows jaahd attaches `NODE_TOKEN` **only** for a path in `BACKEND_CP_ROUTES`, derived (not copied) from `agent_vocabulary._VERBS` + the four broker paths (`cred_broker.py:128-135`) + `POST /api/agent/microvms/{vm_id}/facts` + `GET /api/agent/tasks/{task_id}/attachments/{id}/content`, matched per segment. Backend CP transport selected by `JAAH_CP_TRANSPORT=bridge`; no module branches on OS. A restart of Windows jaahd kills only the client; backend/microvmd/VMs keep running. `JaahService.cs` holds `wsl.exe -d jaah-microvm --exec /bin/sleep infinity` while marked. - **Parity corrections (§2.4):** `.wslconfig` in the `jaah` account's profile **only while marked** (`[wsl2] nestedVirtualization=true`, `memory = host RAM − max(4 GiB, 25%)`, `processors = all`, `vmIdleTimeout=-1`, `[experimental] autoMemoryReclaim=disabled`), removed on mark-off, applied with `wsl --shutdown` at both edges; `free_disk_mib` = min(distro `statvfs`, Windows volume free from the `host` frame); relay own-addresses = distro addrs ∪ `host` frame Windows addrs (via the egress-core child's `host_platform.own_networks()`); token store: `jaah-engine.sh nodelink-apply` binds `/run/jaah-nodelinks` under `/mnt/wsl/jaah-nodelinks`, and the backend's `shared_storage.resolve_shared_auth_dir` resolves the auth NodeLink there (a VM still gets only its one account's token dir). - Facts: the backend reports `runner_clis` via `microvm_images.guest_clis()` through the `facts` frame; Windows jaahd puts that set in its observation while marked (lift the `host_platform.is_linux() and is_marked()` gates at `apps/jaahd/main.py:1173,1196` so a marked Windows node takes its `microvm` block and `runner_clis` from the bridge). File overlap with the macOS-runtime child at those lines: whichever merges second keeps both branches. - **VM signal and kill path (Planner decision — §9.2 omitted it).** Today answer/message delivery into a VM (`reconcile/enact_signal.py:299-309` `_bound_vm`, `:463,515` → `answer_delivery.py:641-722`) and the VM kill (`kill_gates.py:145,216-219`) run in jaahd behind `is_linux()` and read `vms.json`/`jaah-microvmd.sock`, which on Windows live inside the distro. *Decided: these run in the backend, because §2.1 places the Linux MicroVM subsystem there.* Windows jaahd forwards each VM-bound slot intent (a pane-write/answer whose binding is a VM, and a kill whose target has `VM_TARGET_PREFIX`) verbatim as ONE new W→B op `vm_intent {rid, intent}` → reply `{rid, ok}` or the pinned `error` frame (`no_vm` for an unbound/unknown VM, `bad_frame` for a malformed intent), added to `microvm_bridge.py`'s vocabulary; the backend executes it with the existing Linux `enact_signal`/`kill_gates` VM branches (binding via `microvm_registry.bound_to`, unchanged fence). A Windows jaahd restart or drain never kills VMs (the backend keeps them); VM ends arrive only as these intents or the rail. - Docs: the Windows packaging doc and the Windows ADR-0129 per-OS boundary paragraph (§2.5: the Linux boundary inside one more VM — a CH/KVM escape that also bypasses seccomp and landlock reaches the backend's user `jaahd` in the WSL2 utility VM and, through the bridge, only `BACKEND_CP_ROUTES`; a further utility-VM kernel escalation reaches every distro of the `jaah` account and still needs a Hyper-V escape to reach the host; never "stronger"). Acceptance criteria this child owns on Win (PRD §5, verbatim): - AC1 — The mark turns on for a qualifying node (R1). Verify: Open Nodes → Mac (then Win) → MicroVM toggle → turn on. Observe: the toggle stays on, the MicroVM chip shows on the node, and the node's free static worktrees are gone from its worktree list. - AC2 — A non-qualifying node gets a readable, specific refusal (R1). Every state is covered: hypervisor prerequisite unmet, plan off, held worktrees, live VMs while turning off. Verify: (a) On Win with the A2 prerequisite disabled (and on a Mac where the Architect's prerequisite is unmet), turn on → observe a sentence naming that prerequisite; it must not say "needs a Linux node". (b) In an org whose plan and override deny MicroVM → observe "Your plan doesn't include MicroVM nodes." (c) With a user-held worktree → observe the held-worktrees sentence. (d) With a live VM, turn off → observe "Wait for N MicroVM… to end." - AC3 — A fleet task runs end-to-end in a VM (R2, R3). Verify: Create a project whose only node is Mac (MicroVM on). Start a small coder task. Observe: a VM record appears for the task (org MicroVM tab / VM records) and no worktree is leased; the runner works, opens a PR and reaches `done`; the VM record then shows destroyed. Repeat on Win. - AC4 — Advisor and console sessions run in VMs (R2). Verify: On each node, open an advisor chat and a console session for the project. Observe: each gets its own VM record; console attach is interactive and typing works; an answer to a question card reaches the session; ending the console destroys its VM. - AC5 — Guest isolation holds (R3). Verify: In a console VM session on each node, run `uname -s` → `Linux`; `sudo -n true` → fails; `mount` / `ls` at the NodeLink path → no NodeLink mounted. With two concurrent sessions on the same node, a file one writes in its tree is not visible from the other. - AC6 — Egress is enforced (R3). Verify: In a VM session: `curl -sI https://github.com` → succeeds; a host not on the project allowlist → refused; the node's own LAN IP and `169.254.169.254` → refused. Add the host to the project's egress additions (ProjectEditPage) → a new session reaches it. Each refusal is visible in the node's egress refusal log/report. - AC7 — Control calls are bound to the VM's task (R3). Verify: From inside a task VM, call a task-scoped control verb naming another task's id → refused. The runner's own `task_get` returns its own task. - AC8 — The provider set is the Linux guest's (R4). Verify: On Mac (host supports only Claude Code), run a task with a Codex account (and one other non-macOS provider an Ubuntu MicroVM node runs) → it is placed in a VM and runs. On an UNmarked Mac, the same account is still refused with the ADR-0145 sentence. - AC9 — Lifecycle and retention (R3). Verify: Reset a running task → its VM is destroyed. Delete a task → destroyed. Force a task to `system_failed` → its VM shows `retained`, then destroyed after `microvm_retention_hours`. Verify the last step by setting the retention to its minimum on dev, or by reading the lifecycle record. - AC10 — Sizing applies and shows its source (R3). Verify: Set the node size on Node detail (overriding the project size). Start a task → inside the VM `nproc` and `free -m` match the effective size, and the console shows the value with source "node". - AC11 — Placement refusals are readable (R3). Verify: Set a VM size larger than the node's free RAM. Start a task → it waits with the readable placement reason, and no VM is created. - AC12 — Liveness reclaim (R3). Verify: Kill the guest agent session inside a task VM → the attempt ends through broken-session reclaim within the same bound as a Linux node, and the task is redispatched or failed as on Linux. - AC13 — Cloud Windows (R5). Verify: On dev, launch a Windows node through the AWS door on the MicroVM-capable type the Architect names → mark it MicroVM → run AC3 on it → it passes. Launching with MicroVM on an incapable type → readable refusal. - AC14 — Install is turnkey apart from A2 (R6). Verify: Enroll a fresh Mac and a fresh Windows machine with the per-OS one-liner from the console. Perform only the documented prerequisite. Turn MicroVM on → AC3 passes with no other manual step. Update the node package → a subsequent AC3 run still passes. - AC15 — Linux and static behaviour unchanged (R7). Verify: `./dc check full` is green. On dev, a Linux MicroVM node still passes AC3. An unmarked Mac and Win still run a static task as before. Credentials & reach (Architect §8, verbatim rows): - AC1–AC12, AC14 (console flows) | `org/console/dev-login` (operator login), dev console `https://app.dev.jaah.ai` | **verified**: `credential_get` succeeded; the console was opened from this node (`h8agi5dw`), signed in, Nodes list read. The credential's stored `base_url` names the legacy dev host, which does not answer from this node; the oracle uses `app.dev.jaah.ai`. - AC1–AC12, AC14 on Win | Node `win-t8703` (Windows · x64, online, 1.2.4277) on dev | **verified**: read on its Node detail page. AD3 confirms it is the PC. Its Windows edition/build is not shown in the console, so C2's first step is the P2 probe. - AC13 | Dev CP's AWS cloud connection (CP-side; the runner needs only `org/console/dev-login`); a Windows-capable metal type on dev's allow-list | **verified (environment)**: the AWS door is the CP's own cloud connection, and Windows cloud launch is an existing capability (`node_platform.py` `CAP_CLOUD_LAUNCH`, Task 8744). The m8i.large (or fallback m7i.metal-24xl) allow-list entry is made in the console by C2's probe step, as the operator login, which is verified above. No new credential is needed. - AC13 fallback | `m7i.metal-24xl` on dev's allow-list | **deferred**: needed only if the m8i.large probe fails (§5). C2 allow-lists it in the console then. Its Windows capability follows `windows_capable_spec` (x86_64 and metal, `instance_types.py:70-75`), which holds for it by construction. - AC15 Windows static half | `win-t8703` while unmarked | **verified** (row above). C2 runs it before marking. - AC15 gate, AC16, AC17 | Worktree only (`./dc check full`, `tools/check_context.py`, `tools/microvm_pins.py`) | **verified**: nothing beyond the worktree. Out of scope (BA, verbatim — do NOT touch): - Do **not** change Linux MicroVM behaviour, its pins, its egress default list (`apps/api/app/microvm_egress.py`) or its sizing defaults, beyond what is needed to add the OS dimension. - Do **not** change the static worktree pool or dynamic worktrees on any OS. - Do **not** add a new plan term or billing price (D7). - Do **not** mount NodeLinks inside a VM on any OS. - Do **not** add Multi-Node Hosts / co-located nodes (`vnode_host`), GitHub Actions runners or SDK-agent support to macOS or Windows hosts. - Do **not** support Intel Macs, Windows on ARM, or cloud-launched macOS nodes. - Do **not** relax the host-session (unmarked) capability table of ADR-0145. - Do **not** describe the boundary as stronger than each hypervisor gives ("strong isolation", "defence in depth" — ADR-0129 anti-pattern). - No new console page. Existing surfaces only (A4). - Operate and verify Jaah only through the console. No CLI, SQL or cloud-console workaround for live state (CLAUDE.md hard rule). Architect boundaries (declined, do NOT build): a Windows native Hyper-V runtime; Rosetta in Mac guests; boot-latency optimisation (pre-booted snapshots); WSL networking mode stays NAT; no TCP listener or named pipe for the bridge. Also do NOT touch: `jaah-engine` (Alpine) beyond the `nodelink-apply` bind; Linux/macOS runtime code paths; a `.wslconfig` on an unmarked node. Done means (all live runs on dev as the operator, on `win-t8703` unless stated; screenshots/records in the report): - Probe outcome recorded (m8i.large boots or not; Server 2025 WSL KVM yes/no) and `WINDOWS_NESTED_MICROVM` set accordingly with the evidence cited in its docstring. - AC15 Windows static half — BEFORE marking: an unmarked `win-t8703` runs a static task to `done`. - AC1 on Win. AC2(b)(c)(d) on Win (AC2(a) live dropped, AD7). AC3, AC4, AC5, AC6 (incl. the Windows host's own LAN IP refused), AC7, AC9, AC10, AC11, AC12 on Win. - R4 on Windows — an AC8-shaped run: a DeepSeek or Pi account (`windows_runner=False`) task placed in a VM on `win-t8703` runs to `done`; the same account on an unmarked Windows node is refused with the ADR-0145 sentence. - AC13 end to end on the named type (m8i.large or m7i.metal-24xl), plus the incapable-type launch refusal. - AC14 on Win: enroll a fresh Windows machine (a fresh AWS-launched Windows node counts) with the console one-liner, do only the documented prerequisite, mark → AC3 passes; update the node package → AC3 still passes. - Boot-to-runner latency on Win measured vs `gmind-microvm-1`; if > 2×, the reason is recorded. - AC4 answer path on Win: an answer to a question card reaches the VM session (live, part of AC4); unit test: a `vm_intent` carrying an answer for a bound VM reaches the backend's delivery, one for an unbound task/session is refused with `no_vm` (no delivery), and a VM kill intent stops that VM only. - §2.4 parity unit tests: `.wslconfig` written with exactly the pinned keys on mark-on and removed on mark-off (never written on an unmarked node); `free_disk_mib` = min(distro statvfs, `host`-frame volume free) with a 1 TB sparse-vhdx statvfs fixture; own-networks = distro ∪ `host` frame addrs; `nodelink-apply` produces the `/mnt/wsl/jaah-nodelinks` bind and `resolve_shared_auth_dir` resolves there. - R7 on Win live: after the AC runs, mark `win-t8703` off → `.wslconfig` is gone and a static task runs to `done` again. - The Windows bundle CI artifact listing shows the `microvm/` payload (CH, CLOUDHV.fd, virtiofsd, the noble WSL rootfs pin) — cited. - Unit tests for the bridge: frame codec (1 MiB cap), `BACKEND_CP_ROUTES` matching (allowed templates pass; a non-listed path → `forbidden_route` with no CP call), credit window/ack, `hello` proto mismatch → `bad_frame`, unknown op → `unknown_op` without closing. - AC15 — `./dc check full` green (cite GATE_RESULT).