CP shutdown: DBOS workflow persists after DBOS

Task 6880 · Done · Coder · created · updated

On main (1.2.1044), each CP task logs this as ECS stops it during a deploy: ``` [ERROR] (dbos:_core.py:1130) Exception encountered in background workflow: dbos/_core.py:890 persist -> dbos._sys_db.update_workflow_outcome dbos._error.DBOSException: DBOS Error: System database accessed before DBOS was launched ``` Seen at 19:49:41Z and 19:57:10Z on 2026-10-17, from the outgoing task-definition revision 94 tasks, in log group `/ecs/jaah-main-cp`. A background DBOS workflow thread is still finishing when the app's shutdown tears DBOS down, so its outcome is never recorded. Ask: make shutdown drain or stop in-flight DBOS background workflows before `DBOS.destroy()` (or whatever `apps/api/app/durable/` calls). An unfinished workflow is then left to DBOS recovery on the next start, with no error logged and no outcome write lost. Find out which workflow it was and confirm it is recovered. Out of scope: DBOS version bumps unless the fix needs one. Done means: a test reproduces the shutdown-while-workflow-runs ordering and passes; a deploy on dev shows no such error in the stopping tasks' logs; `./dc check full` green.

Outcome

CP shutdown now drains DBOS workflows for up to 3s before destroy; one that outlives the drain logs INFO and stays PENDING for recovery. The workflow was the alarm-relay cron tick. Engine test added, full gate green, PR #5713 merged; dev stops are clean. Follow-ups: draft Task 6910.

Pull request

Artifacts

Activity

  • Created
  • Ready
  • Runner started
  • Runner ended (done)