Add timeout to tailscale join and resolve rate

Task 11248 · Done · Coder · created · updated

Found by the matrix sweep (Task 11113) on dev EC2, 2026-10-29. Four fresh self-hosted installs in the jaah-dev VPC sat at `==> [jaah] joining the tailnet` for more than 15 minutes. tailscaled logged `fetch control key: 429 Too Many Requests` every 15–25 s. The installs finished only after other test machines were terminated. Two causes: - **No timeout.** `tailnet_join_command` (`apps/api/app/node_bootstrap.py`, around line 425) renders `tailscale up …` without `--timeout`. A login that never completes therefore hangs the bootstrap indefinitely. It never prints the `FAILED at step: join the tailnet` line that D17 promises for every step. - **Per-IP limit vs. NAT.** headscale's edge rate-limits per source IP (`infra/jaah/headscale/nginx.conf`: `limit_req_zone … rate=10r/s`, `burst=40` on `/key`). Every machine behind one NAT shares that budget: here about 25 tailscaled clients (hosts plus vnodes) on the dev VPC's NAT. Logged-out clients retry and keep the limit tripped. A customer with many nodes behind one corporate NAT would hit the same thing. Ask: - Give the bootstrap's `tailscale up` a bounded `--timeout`, so a join that cannot complete fails at the named step with tailscaled's last error. - Decide how the edge should treat many nodes behind one IP: key the limit on something other than the source IP for authenticated node traffic, raise the per-IP budget, or exempt the CP's own VPC NAT. Write the decision down. Out of scope: the tailnet ACL, and headscale itself. Done means: - A test proves the rendered join carries `--timeout`, and that a join which never comes up fails with `FAILED at step: join the tailnet` and a non-zero exit. - The edge decision is recorded in `infra/jaah/headscale/README.md`, along with any config change and its plan or apply output.

Outcome

Bootstrap tailnet join bounded by --timeout=300s; a stuck join exits non-zero at "join the tailnet" with tailscaled's status and log. Headscale edge per-IP limit raised to 50r/s burst 200, decision in headscale README. PR #7331 merged (bda73b9) on a green full gate; its deploy was superseded by a newer one. Deferred: WAF rate-limit alarm. Memory: none applied.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)