MicroVM egress relay and DNS

Task 5806 · Done · Coder · created · updated

Epic 5777 — MicroVM nodes. This child is the architecture's **C8: Egress relay**. Depends on the supervisor child (tap pool + nft redirect to `:5353`/`:15001`, §4.3 `egress_refusals` hook) and the images/rail child (per-VM `<vmdir>/allowlist.json` + tap from the provision payload, guest `resolv.conf`/Docker DNS). Upstream: PRD task 5778 `prd`, Architecture task 5779 `architecture`. ## Original requirement (verbatim) Epic Goal 5: "Outbound network from the VM goes only through a host-side egress proxy that enforces a per-task domain allowlist; everything else is unreachable by construction." Epic Replace-or-delete: "The nft-based per-environment egress allowlist: replace with the host egress proxy (domain-based)." Goal 2 (part): "... no NodeLink store, no EFS (port 2049), no instance metadata service." PRD R7: "A guest's only route out is a host-side egress proxy. It admits a connection only to a domain on that task's allowlist: the built-in default (A4) plus the project's own additions minus its removals, edited in the console's project settings. All other destinations (other domains, raw IPs, the node, the tailnet, the metadata service, EFS) are unreachable." R8 (part): "two VMs on one node can each bind the same port." ## Approach (settled) - **D14** in jaahd, `apps/jaahd/microvm_egress.py` (asyncio); no third-party proxy (squid is GPL; allowlist must key on the VM). - **DNS:** each tap's host address `198.19.N.1` answers UDP/TCP 53 (nft-redirected to `:5353`). Name on the VM's effective allowlist (exact or `*.` suffix) → A record from that VM's fake pool `198.18.(N).0/24` (254 names; exhaustion → NXDOMAIN + logged reason), mapping kept per VM for its life. Other names → NXDOMAIN; AAAA → empty NOERROR. - **TCP:** nft redirects every guest TCP flow to `:15001`. Relay reads `SO_ORIGINAL_DST`, maps source tap → VM and fake IP → domain, resolves with the node's resolver, and **refuses when any resolved address is loopback, link-local (169.254/16), RFC1918, CGNAT (100.64/10), the node's own addresses or the VPC CIDR**; then splices to `<real-ip>:<original port>` (any TCP port, Q5). - Each refusal logged with VM, name, reason to `<vmdir>/egress.log` and surfaced in the §4.3 `egress_refusals` tail (last 50, `{vm_id,name,reason,at}`) so the node page shows it. - **Never an open proxy:** listeners bind one socket per tap host address, never `0.0.0.0` (nft input also drops 5353/15001 from non-`jvm*`). - **D16:** Docker in the guest uses the guest resolver (tap host), so containers resolve via fake-IP DNS and leave via guest NAT → tap → relay. No proxy env vars. - Quality: an allowlist enforced by SNI only would pass the curl probe and miss ssh/raw TCP — the fake-IP design is required, and so is the private-range refusal after resolution: an allowlisted name pointing at 169.254.169.254 MUST fail. ## Acceptance criteria (verbatim) - AC-4 Negative isolation (all five must FAIL from inside a running VM). Oracle, from the guest's shell: `ls` / `cat` a sibling task's worktree path and token dir (absent or permission denied); list the NodeLink accounts dir (absent); `nc -zv -w3 <efs-ip> 2049` (fails); `curl -m3 http://169.254.169.254/` (fails); `curl -m5 https://example.org` with example.org not on the allowlist (refused by the proxy). Then add example.org to the project's allowlist in the console, start a new task, and show the same curl succeeds. **Fails if** any probe succeeds, or the allowlisted one does not. (Network probes here.) - AC-7 Docker in the guest. Oracle: in two concurrent VMs on one node, `./dc up` the project's stack on default ports. Both are healthy, with no port-grid offset. (Image pulls through the relay here.) (Here: mechanism + tests; the live run is C12's.) Credentials & reach: tests — none beyond the worktree. Live AC-4/AC-7 on dev — **deferred** to C12 (dev reach Q7). AC-16 — forge via `jaah-forge` — **verified**. Out of scope: no third-party/GPL proxy (N5). No IPv6 egress or UDP beyond DNS (QUIC) — dropped by construction, a named deferred follow-up. Do not change the nft table shape (supervisor child) or the allowlist computation (C2). Do not touch static worktrees, port grid, co-located nodes (N1). Epic: "retiring static worktrees, co-located nodes (Multi-Node Hosts), the port grid or user slots; Mac/Windows hosts; GPU passthrough; any change to pricing beyond moving products to KVM-capable instance families." Done means: - DNS: allowlisted exact and `*.` names get a per-VM fake IP; others NXDOMAIN; AAAA empty; pool exhaustion NXDOMAIN + log — tests. - Relay: fake IP → domain splice on an arbitrary port; refusal for a name resolving to 169.254.169.254, RFC1918, CGNAT, loopback, node address, VPC CIDR — tests (each case). - A connection from tap A cannot use tap B's fake mapping — test. - Listeners bound per tap address only (no `0.0.0.0`) — test. - Refusals appear in `egress.log` and the §4.3 `egress_refusals` tail — test. - `./dc check full` green on the PR; PR auto-merged and the Jaah Deploy for dev succeeds (AC-16).

Outcome

MicroVM egress relay (fake-IP DNS + TCP relay, private-range refusal, refusal log/tail) merged in #5109; dev Jaah Deploy succeeded. Full gate green except 2 docs stages already red on develop (not touched). Follow-ups in report.

Pull request

Artifacts

Activity

  • Draft
  • Ready
  • Runner started
  • Runner ended (done)