Lease drain can never wedge dispatch fleet-wide
Goal: no single stuck cron call can stop dispatch for every org. A firing past its timeout is visible and recoverable from the console. Carries draft 7662.
Evidence (dev, 2026-10-19): unassigning project mr-6382 from MicroVM node gmind-microvm-1 while VMs ran made a Lease drain firing never return. Every later firing was "Skipped — a previous firing had not returned", Task pipeline and Trigger intake hung too, and nothing leased for about 45 min. It survived a redeploy, and the automation still read "Active". Re-creating the cron (new advisory lock) cleared it. It recurred about 10 min later, matching `lease_drain._DESTROY_REMINT_COOLDOWN` in `_destroy_free_dynamic`. `cron_runtime._bounded_call` holds the run lock while the body lives.
1. Reproduce locally (MicroVM node with a live VM for P, unassign P, drain). Find the blocking call (a row lock without lock_timeout in `microvm_lifecycle.converge/sweep`, `slot_allocation._locked_rows`, `destroy_for_task`, or a node-intent wait) and bound or fix it.
2. A firing that outlives its hard timeout records an `error` run, and the automation shows Failing.
3. Operators get a supported recovery that keeps run history, instead of delete-and-recreate.
4. Node › Projects removal names the live tasks/VMs it will kill and asks for confirmation (from 7663 item 8).
Out of scope: Gemini/ACP runtime; the MicroVM "session vanished" crash.
Done means:
- A test reproduces the hang and passes after the fix. A test shows a timed-out firing recorded as an error, with the status reflecting it.
- On dev, unassigning a project from a MicroVM node with a live VM leaves Lease drain producing acted/noop rows for 15+ minutes.
Required checks: ./dc check full
Outcome
Fixed the lease-drain wedge (MicroVM rows lock with SKIP LOCKED; cron lock waits capped at 15s). Stuck schedules show Failing·Stuck with a Release that keeps history. Node removal names the tasks/VMs it kills. Quick gate green; #6190 merged; visual check ok. Dev 15-min check pending deploy. Follow-up draft #7826.
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)