Add timeout to tailscale join and resolve rate
Found by the matrix sweep (Task 11113) on dev EC2, 2026-10-29.
Four fresh self-hosted installs in the jaah-dev VPC sat at `==> [jaah] joining the tailnet` for more than 15 minutes. tailscaled logged `fetch control key: 429 Too Many Requests` every 15–25 s. The installs finished only after other test machines were terminated.
Two causes:
- **No timeout.** `tailnet_join_command` (`apps/api/app/node_bootstrap.py`, around line 425) renders `tailscale up …` without `--timeout`. A login that never completes therefore hangs the bootstrap indefinitely. It never prints the `FAILED at step: join the tailnet` line that D17 promises for every step.
- **Per-IP limit vs. NAT.** headscale's edge rate-limits per source IP (`infra/jaah/headscale/nginx.conf`: `limit_req_zone … rate=10r/s`, `burst=40` on `/key`). Every machine behind one NAT shares that budget: here about 25 tailscaled clients (hosts plus vnodes) on the dev VPC's NAT. Logged-out clients retry and keep the limit tripped. A customer with many nodes behind one corporate NAT would hit the same thing.
Ask:
- Give the bootstrap's `tailscale up` a bounded `--timeout`, so a join that cannot complete fails at the named step with tailscaled's last error.
- Decide how the edge should treat many nodes behind one IP: key the limit on something other than the source IP for authenticated node traffic, raise the per-IP budget, or exempt the CP's own VPC NAT. Write the decision down.
Out of scope: the tailnet ACL, and headscale itself.
Done means:
- A test proves the rendered join carries `--timeout`, and that a join which never comes up fails with `FAILED at step: join the tailnet` and a non-zero exit.
- The edge decision is recorded in `infra/jaah/headscale/README.md`, along with any config change and its plan or apply output.
Outcome
Bootstrap tailnet join bounded by --timeout=300s; a stuck join exits non-zero at "join the tailnet" with tailscaled's status and log. Headscale edge per-IP limit raised to 50r/s burst 200, decision in headscale README. PR #7331 merged (bda73b9) on a green full gate; its deploy was superseded by a newer one. Deferred: WAF rate-limit alarm. Memory: none applied.
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)