Investigate and stop /tmp inode exhaustion on nodes

Task 7006 · Done · Coder · created · updated

Operator ask (in Task 6978, 2026-10-18): "Add a task to investigate why so many inodes." What happened: on the node running Task 6978, `/tmp` (tmpfs) had all 1,048,576 inodes in use while 24G of space was free. Every `mktemp` failed, `./dc check quick` failed in every stage within a second (for every slot on the node), and Claude Code's Bash tool could not write its cwd file. I cleaned it with the operator's approval, deleting only entries older than 60 minutes that carried a known leak signature. Inode use fell from 100% to 9%. Findings (inode counts before the cleanup): 1. `tools/tests/test_merge_lane.py`: about 560k inodes, the largest leak. Its fake forge clones a git repo into `tempfile.mkdtemp()` (line ~177, the merge path; also line ~60) for each simulated PR and merge and never removes it. About 12k directories named `/tmp/<random>/`, each containing `.git` plus `gate.py`/`tests`/`feature_*.txt`. 2. `/tmp/pytest-of-skynet`: about 370k inodes in 309 `pytest-N` run directories. pytest's keep-3 retention is not holding, likely because of parallel or xdist runs per slot. 3. `jaahd-test-nodevault-*` and `jaahd-test-jaah-home-*`: about 14k inodes in about 14k directories from the jaahd suite, never removed. 4. Loose `/tmp/tmp.*` files: about 13k, `mktemp` output from gate scripts with no cleanup trap. Requirements: 1. Confirm each source above and find any others: count new `/tmp` entries per suite or stage over one `./dc check full` run. 2. `test_merge_lane.py`'s fake forge removes every clone it makes (`TemporaryDirectory`, or a `tmp_path`-rooted state dir passed in by the test), and so do the jaahd tests (`tmp_path`, not a bare `mkdtemp`). 3. Gate stages clean up their `mktemp` output (a trap in `tools/check.sh`), and pytest temp retention is bounded per run (`tmp_path_retention_policy`, or a per-run basetemp removed afterwards). 4. The node reports `/tmp` inode use, and the console flags a node above 80%. Out of scope: resizing the tmpfs; cleanup outside `/tmp`. Done means: - One `./dc check full` run leaves no new entries in `/tmp` (show before and after counts), and a test pins that the merge-lane fake forge leaves no clone behind. - `./dc check quick` is green; the lane merges the PR.

Outcome

Stopped /tmp leaks: merge-lane clones, jaahd test dirs, gate output now in a per-run TMPDIR swept when its run dies. Nodes report /tmp inode use; console flags >80%. #5778 merged; hotfix #5793 fixed the migration chain I broke; deploy green.

Pull request Pull request

Artifacts

Activity

  • Created
  • Updated
  • Run
  • Runner started
  • Runner ended (done)