Confirm why ECS replaced the first new
Jaah Deploy run 39044265499 (master 3a3eebd, 1.2.809): the first new `jaah-main-cp:90` task (10.70.10.253) never reported healthy in the `jaah-main-cp` target group during ~60s of polling from +24s, and ECS replaced it about a minute later. The likely cause is a cold start that ran past the 60s `health_check_grace_period_seconds` (infra/jaah/terraform/modules/ecs/main.tf), so the ELB health checks failed. An AZ rebalance is less likely, because a rebalance stops a task that was already healthy. Since Task 6691, infra/jaah/rollout_wait.py prints a `::warning::ECS stopped new task ...` line with the task's stoppedReason and each container's exit reason.
Ask:
- Read the stop reason of a replaced new task, from the next deploy log that has the warning line or from the console's ECS view. If the console cannot show it, add it there instead of using the AWS CLI.
- If the reason is ELB health checks, measure api boot time (migrate + seed + DBOS launch) on main, and either raise the grace period with that measurement as evidence or fix the slow boot.
Out of scope: the deployment circuit breaker settings, the image build, the rollout gate itself.
Done means:
- The stop reason is recorded with the deploy run id that showed it.
- If the cause is a short grace period or a slow boot, a change merges that keeps the first new task from being replaced on cold start, with the measured boot time cited.
Outcome
Raised CP health-check grace from 60s to 120s, based on a measured main boot of 49.8–60.7s over 6 deploys. Run 39044265499's stop reason can't be recovered (ECS drops stopped tasks after ~1h). Full gate green; PR #5630 merged; dev deploy OK. Drafts filed: 6720 (console ECS view), 6721 (boot phase timings).
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)