Jaah Deploy marks main red when ECS

Task 6691 · Done · Coder · created · updated

Jaah Deploy run 39044265499 (master 3a3eebd, 1.2.809) failed `apply (main)` with "the new task never became healthy in the target group within 600s". ECS still finished the rollout. The service event at 11:31 EDT reads "deployment completed", and both running tasks are on `jaah-main-cp:90` with image `3a3eebdc90`. The watcher latched onto the first new task's IP (10.70.10.253). ECS drained and replaced that task about a minute later, so the watcher kept polling a dead target until its deadline. The result is a false red deploy and a page. Make the rollout watcher follow the deployment, not one task IP: re-resolve the new task definition's current tasks each poll, or treat rolloutState COMPLETED on the deployed task definition as success. Also find out why the first task was replaced (a failed health check at cold start?) and record it. Out of scope: the deployment circuit breaker settings, the image build. Done means: - Replaying the sequence (the first new task replaced, then the rollout COMPLETED) passes the watcher in a test. - A crash-looping deployment still fails.

Outcome

Rollout gate moved to infra/jaah/rollout_wait.py; it re-resolves the deployed task def's tasks each poll and logs why ECS replaced any of them. Incident replay passes, crash loops still fail. Full gate passed; PR #5621 merged; dev deploy green. Cold-start grace overrun inferred (58.8s boot vs 60s grace); confirmation is follow-up Task 6695.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)