Debug xdist worker exit code 1
In the full gate (`./dc test -q -n 8`), an xdist worker running a live-DBOS test keeps dying with `node down: Not properly terminated` / `worker exit: exit 1`. There is no `+++ pytest-timeout` dump, no faulthandler dump and no OOM kill. Seen on lane base checks 13453 (develop d584e1fae53f), 13523 (bd642215dd7a, twice) and 13516's predecessor runs, on:
- tests/test_durable_enqueue_engine.py::test_the_same_workflow_id_enqueued_twice_through_the_seam_attaches_to_one_workflow (3 of the last 4 lane runs)
- tests/token_billing/test_outbox.py::test_the_effect_runs_on_dbos_only_when_its_transaction_commits_and_a_failure_is_retried
- tests/test_nodevault_dbos_scan.py::test_every_nodevault_workflow_kind_ran_with_real_crypto
What is known (Task 13541):
- Exit status 1 with no dump matches only `tests/worker_crash.py::_timeout_to_stderr`'s `os._exit(1)`. Nothing in dbos 3.2.0 or app code calls exit. So the test most likely hangs past 120s and the dump is lost; one candidate is the capture manager still holding fd 2 when the timer thread writes.
- Every DBOS suite passes alone, and together under `-n 8` (395 passed; 1365 with the capture suites). Only the full suite reproduces it.
- Task 13385's listener stop (`app/durable/runtime.py::destroy_engine`) and Task 13279's outbox drain are already on develop.
Ask: make the dump reach the gate log (or prove where it goes), then fix the hang it names.
Out of scope: dbos upgrades; the activity-capture loss (fixed in Task 13541).
Done means: the cause of the worker's exit is named with evidence; these tests no longer kill workers across three consecutive full-gate runs; `./dc check quick` green.
Outcome
The worker exit 1 is pytest-timeout's kill. The hangs it named (a DBOS launch blocked by a leaked idle transaction) and the dump channel were already fixed by Tasks 13639/13658. 13 consecutive full-gate runs since have no worker crash; quick gate green. No PR. Memory: none applied.
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)