MicroVM placement, rail and lifecycle

Epic 5777 — MicroVM nodes. This child is the architecture's **C4: CP placement and lifecycle**. Depends on C2 (schema, sizing, allowlist compute). Upstream: PRD task 5778 `prd`, Architecture task 5779 `architecture`. ## Original requirement (verbatim) Epic Goal 1: "Each dispatched task in a `dynamic` project gets a fresh VM with its own kernel, booted from a cached base image plus the project's cached setup, and the VM is destroyed with no trace at `done` / reset / delete (ADR-0129 lifecycle and retention rules unchanged)." (I5: read as "a task allocated to a MicroVM node".) Epic Goal 9 (part): "dynamic tasks are scheduled only onto KVM nodes." PRD R1: "A node has a mode, MicroVM on or off. On a MicroVM node, every session placed there runs in a fresh Cloud Hypervisor microVM with its own kernel: fleet tasks of any persona and project, plus human console sessions and advisor chats. No session on that node runs directly on the node. Every other node behaves exactly as it does today." R2: "... It is destroyed with no trace on the node at `done`, reset and delete. On `system_failed` it is retained per A7. A human or advisor session's VM is destroyed when that session ends." R3 (admission): "A node admits a new VM only when its measured free RAM and disk still clear a reserve after the VM's start-up needs. It never admits on the sum of the caps." R11: "Placement never sends a session to a MicroVM node that has stopped reporting KVM." A3: "Placement picks a node by the existing rules. The chosen node's mode decides how the task runs (I5)." A7: "On `system_failed` alone the VM's disk is kept for the operator-set window, then destroyed. Every other end destroys it at once." ## Approach (settled) - **D21 placement:** `routes/fleet.py::fleet_allocation` stays the single node chooser. For a MicroVM node, `_room_by_node` reports admissible VM count from the node's reported `microvm.free_ram_mib`/`free_disk_mib` (in `nodes.config_snapshot['microvm']`, pinned by C2): `min((free_ram − 2048) // startup_ram, (free_disk − 20480) // startup_disk)`, `startup_ram = min(ram_cap, 2048)`, `startup_disk = 4096` MiB. Because `pool_state.assign_nodes` treats room as a preference, MicroVM nodes with room ≤ 0 or without `microvm.kvm_usable = true` are **filtered out of `live`** before assignment (hard gate). When filtering leaves a task with no node, write the reason into `fleet_allocation`'s existing `waiting_out`: "No MicroVM room: `<node>` needs 2048 MiB free RAM above its 2048 MiB reserve (has 1312)", or "`<node>`'s KVM isn't usable". - `lease_drain` routes each allocation by node mode: static node → unchanged `build_dispatch_plan`; MicroVM node → new `microvm_placement.admit(task, node)`: opens a `microvms` row (sizing from `microvm_sizing.effective`, allowlist from C2's effective compute), picks the account with the same picker the static lease intent uses (any account kind — no API-key-only filter), issues `provision` then `launch`. Script personas get no account. - **§4.2 rail** on `intent_rail` with `rail: "microvm"`: frame `{"rail":"microvm","id","vm_id","intent_seq","generation","verb","payload","reason","created_at"}`. `provision`: `{kind, task_id|null, console_session_id|null, project:{id,slug,clone_url,integration_branch,branch_prefix,push_enabled,stack_up_cmd,stack_down_cmd,has_repo}, branch, sizing:{vcpus,ram_mib,disk_gib}, allowlist:[...], account:{slug, provider, shared:bool}|null}`. `launch`: the same `LAUNCH_KEYS` the static `lease_slot` prompt bundle carries (`prompt_bundle`, `provider`, `persona`, `autoclose`, `runner_session_id`, ...); console sessions `{console_cmd}`; scripts `{kind:"script", tool}`. `destroy`: `{retain:bool}`. Kill intents (`kill_runner`, settled-hold) for a task bound to a VM are translated into `destroy`; the node never sees `kill_runner` for a VM. - **§4.2b facts route:** `POST /api/agent/microvms/{vm_id}/facts {generation, intent_seq, state ∈ provisioning|running|stopped|retained|destroyed|failed, reason|null}` — advance `microvms.state` only for the matching `generation`. On `failed` (e.g. microvmd re-check refused admission): mark row `failed`, return the task to `ready`, copy the sentence into `wait_reason`. Also converge from the `vms` list in the reported block. - **D22:** on a MicroVM node, `routes/console.py` new-session/open creates a `microvms` row of kind `console` (no slot claim, no `worktrees` row) and issues provision + launch with the `clauded_wt` command line; session end (`/exit` or the end route) issues destroy. Advisor chats are runner dispatches and follow D21. - **D23 lifecycle sweep** (`microvm_lifecycle.py`, replacing `env_destroy.py`'s role; do not edit `env_destroy.py` — C1 deletes it): row whose task reaches `done`, is reset or deleted, or whose console session ended → `destroy` at once; task `system_failed` → `destroy{retain:true}`, row `retained`, then `destroy{retain:false}` after `microvm_retention_hours`. Runs inside the existing `lease-drain` pass. - C5c (provision/destroy executor) and C6 (launch) enact this rail in parallel; test both directions against the pinned §4.2/§4.2b shapes with fixtures. ## Acceptance criteria (verbatim, CP halves) - AC-1 End to end ... Observe dispatch → gate green inside the VM → PR auto-merged → task `done` → VM gone ... (CP half: dispatch to the right rail, destroy on done; live run in C12.) - AC-10 (admission): "Run more VMs than the sum of RAM caps would allow. They are admitted while real free memory clears the reserve, and the next is refused with a readable reason once it does not." Credentials & reach: none beyond the worktree for unit/integration tests. AC-16 — forge via `jaah-forge` — **verified**. Live verification is C12's (dev reach deferred, Q7). Out of scope: Do NOT change static dispatch (`build_dispatch_plan`, `lease_slot`), the slot pool machine, `pool_max`, the port grid or co-located nodes (N1) — non-MicroVM nodes byte-for-byte unchanged. Do not modify the old env files C1 deletes. No node-side code (C5–C9). No UI (C3). No backwards-compat shims (N7). Epic: "retiring static worktrees, co-located nodes (Multi-Node Hosts), the port grid or user slots; Mac/Windows hosts; GPU passthrough; any change to pricing beyond moving products to KVM-capable instance families." Done means: - Placement tests: MicroVM node with enough reported free RAM/disk gets the task via `microvm_placement.admit`; with room ≤ 0 or `kvm_usable` false it is filtered and the task's `waiting_out` shows the exact sentence; static nodes unchanged (existing static placement tests still green) — cite. - Admission uses reported free RAM, not the sum of caps — test with caps summing above node RAM. - Rail frames match §4.2 exactly for provision/launch/destroy and console/script variants; `kill_runner` for a VM-bound task becomes `destroy` — tests. - Facts route advances state only on matching generation; `failed` returns task to `ready` with `wait_reason` — tests. - Console session on a MicroVM node creates a `console` row (no worktrees row) and end destroys it — tests. - Lifecycle: done/reset/delete → destroy now; system_failed → retain then destroy after `microvm_retention_hours` — tests. - `./dc check full` green on the PR; PR auto-merged and the Jaah Deploy for dev succeeds (AC-16).

MicroVM C4 shipped: reported-RAM hard placement gate with readable waits, VM admission on new microvm rail (microvm_intents), facts route, kill→destroy, retention sweep, console VMs; 77 new tests, full gate green, PR #5066 merged, dev deploy success. Deferred: Jaah AI account on VMs, C7 attach.