Bastion loses DNS at boot and its SSM agent
After an EC2 reboot of the dev bastion (`jaah-dev-hs-bastion`, i-037b0b8eeda452517, 2026-10-17 09:31 UTC) the console shows the SSM agent start and then hibernate: `lookup ssm.us-east-1.amazonaws.com: i/o timeout` for every attempt in the first minutes, so the host sat in `ConnectionLost` in Systems Manager although the instance was running with both status checks passing. tailscale0 was already up per cloud-init's route table, which points at the tailnet DNS override (`--accept-dns`, headscale's DNS config) taking over resolv.conf before the tailnet is reachable. Find the exact cause (journal of systemd-resolved / tailscaled around boot), make the bastion resolve AWS endpoints with the VPC resolver regardless of tailnet state (e.g. `tailscale up --accept-dns=false` in the bastion user-data, or a resolved split-DNS for the tailnet domains only), apply it to both bastions over SSM, and reboot one to prove the agent reports Online within two minutes.
Out of scope: the headscale server's DNS settings for other clients, the bastion's subnet routes, and the patching timer Task 6630 installed.
Done means:
- After `aws ec2 reboot-instances` on the dev bastion, `aws ssm describe-instance-information` reports `Online` within 2 minutes.
- The fix lives in `infra/jaah/terraform/modules/bastion/scripts/bastion-user-data.sh.tpl` (one home) and the headscale runbook's § Bastion points at it.
Outcome
Cause: tailscaled rewrote resolv.conf to 100.100.100.100 only, which deadlocks DNS at boot. Fix: --accept-dns=false in bastion user-data plus a runbook pointer; applied to both bastions. Dev reboot showed SSM Online in 28s. PR #5595 merged, deploy green. Full gate never fully green in one run (one unrelated API test; passes alone).
Artifacts
Activity
- Created
- Updated
- Run
- Runner started
- Runner ended (done)