Fix cron follow-up pass: DBOS.trigger schedule

Task 5594 · Done · Coder · created · updated

A cron fire that is dropped while a pass is running is supposed to earn one immediate follow-up pass (`apps/api/app/cron_runtime.py::_follow_up_after_release`, line ~303). That follow-up never runs: `DBOS.trigger_schedule` raises "cannot be called from within a workflow", because `_release_then_follow_up` is called from inside the cron tick workflow (`cron_runtime.py:376` in a done-callback, `:408` in a finally). Evidence from CloudWatch over the last 24h: - dev: about 800 warnings a day. - main: about 1,700 warnings a day. - Mostly from the `lease-drain` and `task-pipeline` crons. - The log line is: `WARNING cockpit-cron: cron <id>: the follow-up for a dropped fire did not enqueue (DBOS Error: DBOS.trigger_schedule cannot be called from within a workflow)`. Effect: when a task changes state while a pass is running, the early wake is lost and the task waits for the next interval (up to 10s for lease-drain, 15s for task-pipeline). The comment in `routes/fleet.py::build_dispatch_plan` added by Task 5579 already assumes the follow-up works. 1. Make the follow-up enqueue from outside the workflow context, so `trigger_schedule` is accepted. Examples: hand it to a plain thread or the serving loop after the workflow returns, or use the DBOS-sanctioned in-workflow equivalent if one exists in dbos 3.2.0. Keep "at most one follow-up per pass" and "never raises". 2. Add a test that drives a dropped fire through the real wrapper and asserts that a follow-up tick is enqueued. It must fail on today's code. Out of scope: changing cron cadences, the coalesce window, or the lease-drain/dispatch logic. Done means: - The new test fails before the fix and passes after it. - `./dc check full` is green. - After deploy to dev, CloudWatch `/ecs/jaah-dev-cp` shows zero "follow-up for a dropped fire did not enqueue" warnings over 30 minutes. Run `filter @message like /follow-up for a dropped fire did not enqueue/ | stats count(*)`. - A follow-up actually runs: a debug or info line, or a cron tick within seconds of a dropped fire, is visible in the same log group.

Outcome

Follow-up trigger now runs in an empty contextvars.Context so DBOS accepts it; new test red→green; full gate green; PR #4893 merged + Jaah Deploy OK. Dev: 0 warnings over 17:26–17:59 UTC, follow-up ticks seen after dropped fires. Main pending promotion.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)