Diagnose API xdist workers dying without faulthandler

Task 13326 · Done · Coder · created · updated

Merge-lane task 13310 (tree d4c6bde9e5, artifact gate-1-d4c6bde9e5-red.log) recorded `[gw1] worker exit: killed by signal 11 (SIGSEGV) — a crash in native code; faulthandler's dump is above; memory: oom_kill +0` while `tests/billing/test_subscriptions.py::test_the_remaining_r10_rows` was running. But no faulthandler dump appears anywhere in that log: no `Fatal Python error` line. The lane then filed the victim as a flaky test (Task 13323). That test is a plain HTTP read and passed 30/30 isolated and 3/3 full-file at `-n 8`. Task 12986 assumed every segfault prints a dump, so it read the earlier dumpless crashes as timeouts or outside kills. This crash shows that a SIGSEGV can also die without a dump. A probe in the api container (`python -X faulthandler`, `ctypes.string_at(0)` on a non-main thread) DOES print a dump under 3.14t, showing only the current thread ("Cannot show all threads while the GIL is disabled"). So this crash is one where the handler could not run, or where its output went somewhere other than the gate log. Candidates: a C stack overflow on a thread with no alternate signal stack, a crash inside the source-built psycopg C extension (no upstream free-threaded wheel), or the worker's stderr fd being redirected at crash time. Ask: - Make a dumpless SIGSEGV diagnosable. For example, enable core dumps or a `catchsegv`-style native backtrace for the api test container in the lane, or prove where the faulthandler output goes in an xdist worker. - Find and fix the native crash. If that cannot be done yet, have the lane classify a SIGSEGV with no faulthandler dump as infrastructure and stop filing the running test as a flake. - Fix the `worker_crash.py` SIGSEGV hint ("faulthandler's dump is above"), which is false in this case. Out of scope: editing the victim tests; reverting the free-threaded runtime without evidence. Done means: a reproduction or a lane log that names the crashing native frame, plus either the cause fixed or dumpless segfaults no longer filed as flaky tests against unrelated tests.

Outcome

Rebased PR #7873 onto develop, resolving the conflict with Task 13328's overlapping fault-log change by keeping this task's superset (per-thread altstacks, DUMPLESS_SEGV, lane classification). Quick gate green; re-queued in merge lane.

Pull request

Artifacts

Activity

  • Created
  • Run
  • Runner started
  • Runner ended (done)
  • Run
  • Runner started
  • Runner ended (done)