Resolve sustained OOM loop on jaah-node-pool-c-host

Task 12152 · Done · Coder · created · updated

Found by Task 12144 (Security Hub review) while tracing Security Hub SSM.3 on the hosting account (664831647587): the node host `jaah-node-pool-c-host` (EC2 i-0b17c7e4d69e331e2, m8i.large, 8 GB, launched 2026-10-11, tag jaah:node=pool-c-host) is starved of memory. Its EC2 console output on 2026-11-01 shows a continuous out-of-memory loop: the kernel repeatedly kills python3 workers of four slot users (UIDs 600500, 600700, 602200, 609800, each ~45 MB RSS), and has killed amazon-guardduty-agent, tailscaled and apt-cache; the SSM agent logs "unable to acquire credentials ... context deadline exceeded" and has reported PingStatus ConnectionLost since 2026-10-30 (last ping 2026-11-01 08:32 UTC), so its GuardDuty and Inspector plugin associations fail with DeliveryTimedOut. EC2 status checks pass and CPU is ~25% average, so the host looks healthy from outside. The sibling host jaah-node-gmind-m8i-1-host (same AMI, same size) is fine. Consequences: runtime threat detection (GuardDuty Runtime Monitoring) is blind on that host, the host is unreachable over SSM, the tailnet link flaps, and whatever tasks run in those four slots are being killed mid-run. Ask: size the node to its slots (a bigger instance type, or fewer worktrees per node, in the node group's capacity rules — apps/api/app/node_group_reconcile.py), and consider a per-slot memory ceiling (a systemd slice or cgroup limit per slot user) so one task cannot starve the host's agents. Risk register row R-SOC-63 tracks it (docs/compliance/risk-register.md). Out of scope: Security Hub rules or suppressions (SSM.3 must clear by the host becoming healthy); the GuardDuty findings themselves; any change to the other hosting nodes. Done means: - `aws ssm describe-instance-information` in the hosting account reports pool-c-host (or its replacement) PingStatus Online and AssociationStatus Success, and Security Hub SSM.3 shows PASSED for it. - The host's console output shows no "Out of memory: Killed process" line in the 24 hours after the change. - The chosen ceiling (instance size, worktrees per node, or per-slot memory limit) is declared in code or the console, not set by hand on the instance.

Outcome

Vnode units + vnode user@ now in capped jaahvnodes.slice (Mem 80/85%, Tasks 80%; user@ TasksMax 33%). Quick gate + 3 review lanes green; PR #7572 merged (7146a39f36), Jaah Deploy success. Deferred: pool-c-host needs a reboot to take it (no console action, draft 12173); live SSM/24h checks unverified. No UI. Memory: none applied; proposed 1 gotcha.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)