Investigate API pytest xdist worker crashes
Merge-lane full gates on 2026-11-02 show API unit (`./dc test -n 8`) workers dying mid-test with `[gwN] node down: Not properly terminated` / `worker 'gwN' crashed while running '<test>'` — no assertion, no pytest-timeout dump. Each time the named test is unrelated and passes on rerun, so the lane files a "flaky test" task per victim (Tasks 12903, 12975 were such; both found nothing wrong in the test).
Seen in: lane 12880 (tests/test_api_keys.py, tests/test_pending_org_gate.py, two workers in one run), lane 12900 (tests/billing/test_subscriptions.py), lane 12965 (tests/test_org_setup.py::test_add_slots_ignores_assignments_that_cannot_run_work). All three are after the free-threaded CPython 3.14t runtime merged (PR #7759, Task 12733, 2026-11-02 12:17 -05:00); no earlier report on this node carries the signature. Correlation only — not proven.
Ask:
- Make a worker death diagnosable: get the dying worker's signal/exit status and any faulthandler output into the gate log (worker stderr is not in it today).
- Find the cause (candidates: a segfault in a source-built C extension under 3.14t, the host OOM killer, the 120s pytest-timeout thread method) and fix it, or make the lane classify a worker crash as infrastructure rather than filing a flaky-test task against the victim.
Out of scope: changing the victim tests; reverting the free-threaded runtime without evidence.
Done means: a lane gate log that shows why a worker died (or a reproduction of the crash), and either the cause fixed or crashes no longer filed as flaky tests against unrelated tests.
Outcome
Dead xdist workers now log their exit signal, OOM counters and timeout stack dump; the lane files no flake for an outside-killed worker. Cause narrowed to a 120s hang or an outside kill (segfault ruled out), not proven. PR #7852 merged; deploy superseded.
Artifacts
Activity
- Created
- Run
- Runner started
- Runner ended (done)