Investigate and stop /tmp inode exhaustion on nodes
Operator ask (in Task 6978, 2026-10-18): "Add a task to investigate why so many inodes."
What happened: on the node running Task 6978, `/tmp` (tmpfs) had all 1,048,576 inodes in use while 24G of space was free. Every `mktemp` failed, `./dc check quick` failed in every stage within a second (for every slot on the node), and Claude Code's Bash tool could not write its cwd file. I cleaned it with the operator's approval, deleting only entries older than 60 minutes that carried a known leak signature. Inode use fell from 100% to 9%.
Findings (inode counts before the cleanup):
1. `tools/tests/test_merge_lane.py`: about 560k inodes, the largest leak. Its fake forge clones a git repo into `tempfile.mkdtemp()` (line ~177, the merge path; also line ~60) for each simulated PR and merge and never removes it. About 12k directories named `/tmp/<random>/`, each containing `.git` plus `gate.py`/`tests`/`feature_*.txt`.
2. `/tmp/pytest-of-skynet`: about 370k inodes in 309 `pytest-N` run directories. pytest's keep-3 retention is not holding, likely because of parallel or xdist runs per slot.
3. `jaahd-test-nodevault-*` and `jaahd-test-jaah-home-*`: about 14k inodes in about 14k directories from the jaahd suite, never removed.
4. Loose `/tmp/tmp.*` files: about 13k, `mktemp` output from gate scripts with no cleanup trap.
Requirements:
1. Confirm each source above and find any others: count new `/tmp` entries per suite or stage over one `./dc check full` run.
2. `test_merge_lane.py`'s fake forge removes every clone it makes (`TemporaryDirectory`, or a `tmp_path`-rooted state dir passed in by the test), and so do the jaahd tests (`tmp_path`, not a bare `mkdtemp`).
3. Gate stages clean up their `mktemp` output (a trap in `tools/check.sh`), and pytest temp retention is bounded per run (`tmp_path_retention_policy`, or a per-run basetemp removed afterwards).
4. The node reports `/tmp` inode use, and the console flags a node above 80%.
Out of scope: resizing the tmpfs; cleanup outside `/tmp`.
Done means:
- One `./dc check full` run leaves no new entries in `/tmp` (show before and after counts), and a test pins that the merge-lane fake forge leaves no clone behind.
- `./dc check quick` is green; the lane merges the PR.
Outcome
Stopped /tmp leaks: merge-lane clones, jaahd test dirs, gate output now in a per-run TMPDIR swept when its run dies. Nodes report /tmp inode use; console flags >80%. #5778 merged; hotfix #5793 fixed the migration chain I broke; deploy green.
Artifacts
Activity
- Created
- Updated
- Run
- Runner started
- Runner ended (done)