Hermetic environments on Cloud Hypervisor: one microVM per task

Task 5777 · Done · Epic · created · updated

Rebuild the dormant hermetic-environment path (ADR-0129, `HERMETIC_ENVIRONMENTS_ACTIVE = False`) so that a task in a `dynamic` project runs inside its own Cloud Hypervisor microVM, and switch it on. No spike first: build it to work. Static worktrees, co-located nodes and the port grid are untouched by this epic; retiring them is a later decision. Background: the attached report (task 5720) compares Devin, Factory, T3 Code and Obvious with Jaah. Devin is the only one with a hermetic machine per agent; Jaah's per-task boundary today is an OS user on a shared kernel, defeated by a kernel privilege escalation (ADR-0129), and the path is switched off. ## Goals (what must be true when this ships) 1. Each dispatched task in a `dynamic` project gets a fresh VM with its own kernel, booted from a cached base image plus the project's cached setup, and the VM is destroyed with no trace at `done` / reset / delete (ADR-0129 lifecycle and retention rules unchanged). 2. The VM sees only what its task needs: its own worktree, its own config dir, and the token directory of the ONE account it runs under. Nothing else on the host, no sibling task, no other account, no NodeLink store, no EFS (port 2049), no instance metadata service. 3. NodeLink works: a NodeLink subscription OAuth account can run the task, the CLI's refreshed token is written back to the org's store, and the refreshed token is visible from other nodes. The current API-key-only limit on environments is removed. 4. Task identity comes from the VM itself (the per-VM vsock socket jaahd created), never from the guest's own claims, a uid, or process ancestry. Every control call (vocabulary tools, credential broker, jaah-forge, git askpass) works unchanged from inside the guest. 5. Outbound network from the VM goes only through a host-side egress proxy that enforces a per-task domain allowlist; everything else is unreachable by construction. 6. Docker Compose (the project's own `./dc` stack) runs inside the VM on default ports. 7. The operator console can attach to the running task's terminal, and answer delivery to a waiting question works. 8. The gate commands run inside the VM; the task's commits reach `origin` and the push guards and hooks still apply. The leave-nothing-behind settle check still holds. 9. Nodes report whether they have KVM; cloud node launches enable nested virtualization; hosting products move to KVM-capable Intel 7th/8th-gen families (m7i/m8i and c/r equivalents; t3, m6i and Graviton do not qualify); dynamic tasks are scheduled only onto KVM nodes. 10. Everything the node needs comes from the `jaah` package (pinned `cloud-hypervisor`, `virtiofsd`, firmware) plus images the node downloads itself (stock Ubuntu cloud image, built into the base image on the node with cloud-init). No GPL binaries shipped in our package. 11. ADR-0129 is amended to say the boundary is the VM, with the attack that defeats it stated in one sentence. ## Replace or delete (do not keep these for their own sake) - `envhelper` root helper and its six operations, the per-environment uid/subuid allocation, netns/veth/nft per environment, `env_registry` (uid→task) and the `SO_PEERCRED` identity path for environments: delete or replace with the VM driver. Keep a per-VM unprivileged user or seccomp/landlock only as VMM-escape hardening if the architecture stage judges it worth it, not for identity. - `jaah-envd@` (rootless Docker per environment) and `jaah-env@` units: delete; Docker runs as ordinary Docker inside the guest. - Any guest agent, custom guest kernel or image pipeline: do not build; prefer stock cloud image + cloud-init + systemd units (autologin to `launch.sh`, socket forwards over vsock). - tmux: not in the guest. On the host it is at most a thin adapter attaching to the VM's PTY; the architecture stage decides whether to keep that adapter for this epic or have jaahd own the PTY (ring buffer + console websocket) and drop tmux for environments entirely. - The nft-based per-environment egress allowlist: replace with the host egress proxy (domain-based). - The API-key-only account restriction on environments: remove. ## Open decisions for the analysis stages (decide, do not ask unless autonomous mode is OFF) - Worktree inside the VM: virtio-fs share from the host (keeps host-side git/guards/hooks) vs. a clone on the VM disk synced over vsock. Pick on measured gate time under nested virtualization; the first is preferred if fast enough. - Base image caching per node and per project: cold-start target and disk budget. - Console attach: host tmux adapter vs. jaahd-owned PTY (see above). - Mac/Windows self-hosted nodes are out of scope here; dynamic mode simply requires a Linux KVM node. Out of scope: retiring static worktrees, co-located nodes (Multi-Node Hosts), the port grid or user slots; Mac/Windows hosts; GPU passthrough; any change to pricing beyond moving products to KVM-capable instance families. Done means: - A real task in a `dynamic` project completes end to end (dispatch → gate green → auto-merged PR → `done` → VM destroyed, nothing left) on BOTH a nested-virtualization m8i cloud node AND a physical KVM host, using a NodeLink subscription account; the refreshed token is observed from a second node. - From inside a running VM, each of these is demonstrated to FAIL: reading a sibling task's worktree or token dir, listing other accounts, connecting to port 2049, connecting to 169.254.169.254, reaching a domain not on the allowlist. - Console attach to the running task and delivery of an answer to a waiting question are demonstrated. - Measured and reported against a static worktree on the same node: cold start, `./dc check full` wall time, resident memory per task. - ADR-0129 amended; hosting products updated; `HERMETIC_ENVIRONMENTS_ACTIVE` is True and a project can enter `dynamic` mode from the console. - Required checks: ./dc check full

Subtasks

  • [business_analysis] Hermetic environments on Cloud Hypervisor: one microVM per task Done
    Settled MicroVM requirements after a 13-answer interview: a node marked MicroVM (plan feature, co-located OK) runs every session in its own Cloud Hypervisor VM; slot_mode=dynamic and the old env code are deleted; perf benchmarks out of scope. 17 ACs in the PRD.
  • [architecture] Hermetic environments on Cloud Hypervisor: one microVM per task Done
    Designed MicroVM nodes for a Cloud Hypervisor VM per session; spec-validator passed on the third round. Contracts are pinned in the architecture document. 12 children: removal, CP model, UI, placement, VM runtime, launch/identity, PTY, egress, co-located nodes, launches/products, ADR, dev e2e. Dev-console oracles deferred: dev refuses this node until fleet egress is added (Q7).
  • [plan] Hermetic environments on Cloud Hypervisor: one microVM per task Done
    14 coders on the Architect's C1–C12 seams (C5 split 3-way), 2 phases. Pinned pool-tick mark + config-snapshot facts. No Required-checks line (none fires on develop PRs). Prod catalogue column deferred to promotion. develop gate red: tenancy census.
  • MicroVM control-plane model and API Done
    MicroVM C2 shipped: migration, sizing/egress/mode modules, node/project/org/plan REST, pool-tick + snapshot seams; 61 tests; quick gate + CI green; PR #5052 merged, dev deploy success. Deferred: converge static_target on marked nodes (C4), C8 private-IP refusal.
  • MicroVM binary pins and licence gate Done
    MicroVM pins (CH v53.0, CLOUDHV.fd, noble virtiofsd; node-fetched noble image + Node 24) staged into the .deb with digest refusal; GPL licence gate; new Depends. #5037 + test fix #5046 merged; dev Jaah Deploy succeeded.
  • Nested virt launches and KVM products Done
    Nested virt on all launch paths, microvm_capable on the API, a KVM-only fresh-DB seed, and doc updates. Merged in #5042, with #5057 adding a review fix and a pr-review cron fix. Prod and dev now offer only KVM (m8i) products. Jaah Deploy dev is green. Deferred: the prod MicroVM column waits for master promotion; UX suggestions are in the report.
  • Remove the old environment path Done
    Old env path removed: envhelper→rootlib, env daemon/CP modules, slot_mode, env units, API-key env rule; drop migration 202610112100. Full gate green; PR #5068 merged, dev deploy success. Deferred: host senv-* purge; nodelink.md:142 for C11 (mem-2).
  • MicroVM console settings and panels Done
    MicroVM console UI: node tab (mark switch, size, full report), project size + egress editor, org tab, plan/override switches, catalogue column. Gate stages green (one develop-only e2e flake). PR #5095 merged; dev deploy OK. UX follow-ups in report.
  • MicroVM placement, rail and lifecycle Done
    MicroVM C4 shipped: reported-RAM hard placement gate with readable waits, VM admission on new microvm rail (microvm_intents), facts route, kill→destroy, retention sweep, console VMs; 77 new tests, full gate green, PR #5066 merged, dev deploy success. Deferred: Jaah AI account on VMs, C7 attach.
  • MicroVM supervisor and tap network Done
    MicroVM tap pool + nft table (root converge), jaah-microvmd supervisor, §4.3 facts, units. Full gate green; PR #5065 merged, dev Jaah Deploy succeeded. K2/D19b proven on pinned CH; K1 static, real boot pending C12. Follow-ups in report (mem-6603, mem-6604).
  • MicroVM session launch and identity Done
    C6 merged (#5110): VM session prepare/launch, VmCaller identity, K6 owner map, guard protection. CI green; dev deploy green (f62a283). Full gate: 2 pre-existing docs stages red (untouched files). Live ACs deferred to C12.
  • MicroVM egress relay and DNS Done
    MicroVM egress relay (fake-IP DNS + TCP relay, private-range refusal, refusal log/tail) merged in #5109; dev Jaah Deploy succeeded. Full gate green except 2 docs stages already red on develop (not touched). Follow-ups in report.
  • Co-located MicroVM nodes Done
    C9: per-vnode /dev/kvm+virtiofsd ACL (daemon uid only) and jaah-microvm@/converge@/microvmd@ units in the vnode netns; K7 relaxed on microvmd@ only. PR #5100 merged via lane; dev Jaah Deploy succeeded. Live run deferred to C12; follow-ups in report (mem-7061).
  • Rewrite ADR-0129 for MicroVM boundary Done
    ADR-0129 rewritten for the VM boundary (one attack sentence); G1/G3 + cross-refs restated; new microvm-nodes.md routed from hub. check_context green, AC-12 grep clean. PR #5113 merged; dev Jaah Deploy success. Deferred: stale code strings, offboard VM disks.
  • MicroVM end-to-end acceptance on dev Done
    Dev MicroVM C12: fixed 5 live blockers (#5127, #5139, #5150; dev deploys green). A task VM now boots on m8i vn5862b and tears down clean. AC-11/12/13/15 pass. Deferred: AC-8 VM attach (F1), repo PR path (no dev GitHub launcher), physical host, in-guest probes.
  • [finish] Hermetic environments on Cloud Hypervisor: one microVM per task Done
    All 13 children done, but the epic's end-to-end Done-means is unmet: VM tasks have no terminal and no gate→PR→done run in a VM yet. Filed 5955 (attach), 5956 (liveness), 5957 (e2e acceptance), 5958 (UX).
  • MicroVM task: end an attempt whose guest Done
    MicroVM task liveness: a guest heartbeat feeds node guest_sessions; the CP ends a dead or never-started VM attempt through the existing reclaim and destroys the VM. Full gate green, PR #5166 merged, dev deploy green. Dev kill test left to 5957.
  • MicroVM console UX fixes found during acceptance Done
    Fixed all 9 MicroVM console findings (PRs #5165, #5178, merged; dev deploys green). Items 1,4–8 shown live on dev; 2,3,9 test-proved. Full gate green bar develop-red tests from other tasks. Follow-ups named in report (task create refused, R30).

Artifacts

Attachments

Activity

  • Created
  • Updated
  • Runner started
  • Ready
  • Ready
  • Ready
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Runner started
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Draft
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Ready
  • Runner ended (done)
  • Runner started
  • Runner started
  • Runner started
  • Runner ended (done)
  • Runner started
  • Runner ended (done)
  • Runner started
  • Runner started
  • Runner ended (done)
  • Runner started
  • Runner ended (done)
  • Runner ended (done)
  • Runner ended (done)
  • Runner started
  • Runner ended (done)
  • Runner started
  • Runner started
  • Runner ended (done)
  • Runner ended (done)
  • Runner ended (done)
  • Runner started
  • Runner started
  • Awaiting response
  • Awaiting response
  • Awaiting response
  • Runner resumed
  • Runner ended (done)
  • Ready
  • Runner started
  • Runner ended (done)
  • Runner started
  • Ready
  • Ready
  • Runner ended (done)
  • Runner started
  • Runner started
  • Runner ended (done)
  • Runner ended (done)
  • Runner ended (done)