Bastion loses DNS at boot and its SSM agent

Task 6638 · Done · Coder · created · updated

After an EC2 reboot of the dev bastion (`jaah-dev-hs-bastion`, i-037b0b8eeda452517, 2026-10-17 09:31 UTC) the console shows the SSM agent start and then hibernate: `lookup ssm.us-east-1.amazonaws.com: i/o timeout` for every attempt in the first minutes, so the host sat in `ConnectionLost` in Systems Manager although the instance was running with both status checks passing. tailscale0 was already up per cloud-init's route table, which points at the tailnet DNS override (`--accept-dns`, headscale's DNS config) taking over resolv.conf before the tailnet is reachable. Find the exact cause (journal of systemd-resolved / tailscaled around boot), make the bastion resolve AWS endpoints with the VPC resolver regardless of tailnet state (e.g. `tailscale up --accept-dns=false` in the bastion user-data, or a resolved split-DNS for the tailnet domains only), apply it to both bastions over SSM, and reboot one to prove the agent reports Online within two minutes. Out of scope: the headscale server's DNS settings for other clients, the bastion's subnet routes, and the patching timer Task 6630 installed. Done means: - After `aws ec2 reboot-instances` on the dev bastion, `aws ssm describe-instance-information` reports `Online` within 2 minutes. - The fix lives in `infra/jaah/terraform/modules/bastion/scripts/bastion-user-data.sh.tpl` (one home) and the headscale runbook's § Bastion points at it.

Outcome

Cause: tailscaled rewrote resolv.conf to 100.100.100.100 only, which deadlocks DNS at boot. Fix: --accept-dns=false in bastion user-data plus a runbook pointer; applied to both bastions. Dev reboot showed SSM Online in 28s. PR #5595 merged, deploy green. Full gate never fully green in one run (one unrelated API test; passes alone).

Pull request

Artifacts

Activity

  • Created
  • Updated
  • Run
  • Runner started
  • Runner ended (done)