Lease drain can never wedge dispatch fleet-wide

Task 7781 · Done · Coder · created · updated

Goal: no single stuck cron call can stop dispatch for every org. A firing past its timeout is visible and recoverable from the console. Carries draft 7662. Evidence (dev, 2026-10-19): unassigning project mr-6382 from MicroVM node gmind-microvm-1 while VMs ran made a Lease drain firing never return. Every later firing was "Skipped — a previous firing had not returned", Task pipeline and Trigger intake hung too, and nothing leased for about 45 min. It survived a redeploy, and the automation still read "Active". Re-creating the cron (new advisory lock) cleared it. It recurred about 10 min later, matching `lease_drain._DESTROY_REMINT_COOLDOWN` in `_destroy_free_dynamic`. `cron_runtime._bounded_call` holds the run lock while the body lives. 1. Reproduce locally (MicroVM node with a live VM for P, unassign P, drain). Find the blocking call (a row lock without lock_timeout in `microvm_lifecycle.converge/sweep`, `slot_allocation._locked_rows`, `destroy_for_task`, or a node-intent wait) and bound or fix it. 2. A firing that outlives its hard timeout records an `error` run, and the automation shows Failing. 3. Operators get a supported recovery that keeps run history, instead of delete-and-recreate. 4. Node › Projects removal names the live tasks/VMs it will kill and asks for confirmation (from 7663 item 8). Out of scope: Gemini/ACP runtime; the MicroVM "session vanished" crash. Done means: - A test reproduces the hang and passes after the fix. A test shows a timed-out firing recorded as an error, with the status reflecting it. - On dev, unassigning a project from a MicroVM node with a live VM leaves Lease drain producing acted/noop rows for 15+ minutes. Required checks: ./dc check full

Outcome

Fixed the lease-drain wedge (MicroVM rows lock with SKIP LOCKED; cron lock waits capped at 15s). Stuck schedules show Failing·Stuck with a Release that keeps history. Node removal names the tasks/VMs it kills. Quick gate green; #6190 merged; visual check ok. Dev 15-min check pending deploy. Follow-up draft #7826.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)