CP deploys: target healthy fast, no abandoned
Goal: a control-plane rollout reaches healthy close to the health-check floor, and a CP shutdown or rolling deploy never leaves a durable workflow PENDING until the next deploy. The items below are goals, not literal steps.
Rollout time
Task 6721's boot timings (Jaah Deploy run 39060651014) show the api ready in 16.7s, then 51.8s from "app ready" to "healthy in the TG". The health-check floor is 5–15s.
1. Find where the 51.8s goes. Candidates:
- TG registration happening only once the whole task is RUNNING;
- the nginx sidecar's readiness;
- `rollout_phases.py`'s inferred healthy mark trailing the real one. `rollout_wait.py` printed TARGET HEALTHY at 46 vs tasks RUNNING at 68.
Measure directly with ELB `describe-target-health` timestamps or TG events. Fix the dominant cause, re-measure, and update the numbers in the `health_check_grace_period_seconds` comment in modules/ecs/main.tf.
2. `apps/api/entrypoint.sh`'s single-worker guard imports `app.routes.tasks` in a separate interpreter (~4s of every boot). Make the guard avoid the app import graph.
Durable workflows at shutdown (follow-ups from Task 6880)
3. Shorten the `alarm-relay` cron's ~10s SQS long poll (30s interval), or end it at shutdown. Today it is in flight on almost every CP shutdown, and the 3s drain (`SHUTDOWN_DRAIN_SEC`) cannot record it.
4. Recover an abandoned workflow without waiting for the next deploy. Every ECS task shares DBOS executor id `local`, and DBOS recovers only at launch. Consider per-task executor ids, plus a periodic recovery of executors whose task is gone (`cp_ecs_tasks` already records the CP's ECS tasks).
Out of scope: the health-check grace value (Task 6695); the circuit breaker; the image build; the shutdown drain itself (Task 6880); DBOS version bumps.
Done means:
- A Jaah Deploy log's `rollout api boot` notice shows "app ready -> healthy in the TG" near the 5–15s floor, or names exactly why it cannot be.
- A CP deploy on dev leaves no `sched-cron:<alarm-relay>` row PENDING after the old task stops, or one left PENDING is recovered without another deploy.
- `./dc check full` is green.
Consolidates drafts 6824 and items 2–3 of 6910 (Task 6937).
Outcome
Merged #5737 and #5754, and the dev deploy succeeded: app ready -> healthy in the TG now reads 0.4s (it was a 51.8s inferred mark), boot guard 4s -> 0.2s, alarm-relay poll 1s, per-task DBOS executors with recovery of stopped tasks' work. The dev recovery check is blocked on the org/ boundary apply (draft #7028).
Artifacts
Activity
- Run
- Runner started
- Awaiting response
- Awaiting response
- Runner resumed
- Runner ended (done)