CP disk gate and events

Epic: Node disk self-healing — no node ever fills (Task 5761). You build child C1 of 6: the control-plane half — disk gate with hysteresis, disk events table + API, alert, gha CI pause, warn threshold served to nodes. ## Original requirement (verbatim, epic) - G2. "When a node does get close, nothing stalls: no runner or CI job starts on a disk that cannot hold it, and the node recovers on its own." - G3. "The operator can see it happen: usage, what was reclaimed, and why a node was paused." - D7. "Thresholds are the existing ones (`apps/api/app/node_host_metrics.py`: warn 75%, critical 90%). At warn the node reclaims; at critical the scheduler stops dispatching runners and CI jobs to it and the operator is notified; dispatch resumes by itself below warn." - Done means (epic): "A synthetic fill past 75% triggers reclaim within one cycle and the console shows the event; past 90% the node gets no new runner or CI job until it drops below 75%, then resumes unattended." ## PRD requirements (verbatim, R7 + R8.3) R7 — Dispatch gate at critical, with hysteresis (D7, G2). States for one node's disk reading: | Reading | Node was not disk-paused | Node was disk-paused | | ok (< 75%) | dispatch normal | resume: runners + CI accepted again, unattended | | warn (75–<90%) | dispatch normal; reclaim runs | stays paused (hysteresis) | | critical (≥ 90%) | pause: no new runner launch, CI runners stop accepting jobs; operator notified | stays paused | | unknown (stale / missing / unreadable) | fail OPEN: dispatch normal (same rule as launch_disk.py) | keeps the last decided state; never resumes on an unknown reading | - AC-7.1 Past 90%, no fleet runner is launched or resumed on the node; a queued task waits with a visible reason naming disk usage, not a silent stall. - AC-7.2 Past 90%, the node's idle CI runners accept no new GitHub job; a job already running finishes (never killed). - AC-7.3 When the reading drops below 75%, runner dispatch and CI runners resume with no operator action, within one reclaim/observe cycle. - AC-7.4 Between 75% and 90% after being paused, the node stays paused. - AC-7.5 A pool the operator paused by hand stays paused after disk recovery; disk-recovery resumes only what the disk gate itself paused. - AC-7.6 On a Multi-Node Host, every node whose own reading is critical is paused (co-located vnodes share the filesystem, so a full host pauses all of them). - AC-7.7 The existing runner_launch_disk_floor_gb floor and the new percentage gate must not present two contradictory answers for one decision: the console shows ONE reason for a node not launching, and the docs state how the two relate (fold them into one rule, or document why both exist). - AC-8.3 The operator is notified when a node enters critical/disk-paused, through the console's existing operator notification channel (no new channel). ## Approach (settled by the Architect — realize, do not reopen) State: new columns on `nodes`: `disk_gate text NOT NULL DEFAULT 'open' CHECK (disk_gate IN ('open','paused'))`, `disk_gate_changed_at timestamptz NOT NULL DEFAULT now()`. Additive migration. Transition — in `fleet_observe.apply_observation`, right after the host_metrics upsert, only when the observation carries a disk reading (disk_total_kb > 0, disk_free_kb ≥ 0); pct = `node_host_metrics.disk_used_pct`: - open and pct ≥ CRITICAL_PCT → paused; insert `pause` event; `record_alert(kind='node_disk_paused', node_id, reason='disk 92% used …')`; poke `gha_pool.enqueue_apply`. The alert's dedup identity MUST include the node: `uq_alerts_dedup` does not cover node_id (`alert_record.py:157`), so put the node id into a field the key covers (e.g. `issue=f"disk:{node_id}"`). A test pins that two nodes pausing in the same minute produce two alerts. - paused and pct < WARN_PCT → open; insert `resume` event; poke enqueue_apply. - anything else, and every observation with no/unreadable disk → no change. Launch gate fold (AC-7.7): `launch_disk.py` stays the ONE home of "may a runner launch on this node, disk-wise". Rule: blocked when disk_gate = 'paused' OR (fresh reading AND free < floor). `wait_reason` returns ONE sentence, pause first: new `dispatch_wait.disk_paused(pct)` → "Waiting: node disk 92% full; resumes under 75%"; else the existing low_disk sentence. Rename `names_below_floor` → `names_blocked` (same callers: lease_drain, routes/fleet.py). Module docstring states how the two relate: % gate = fleet hygiene with hysteresis that also stops CI; GB floor = physical minimum one launch needs on any disk size. Keep both inputs, one rule, one reason. CI gate: the agent gha desired endpoint (`routes/agent.py` ~3711-3719) serves `state = 'paused' if pool.desired_state == 'running' and node.disk_gate == 'paused' else pool.desired_state`, plus `paused_by: 'operator' | 'disk' | null`. The existing drain (busy slots excluded from the stop list) then stops only idle runners. The operator's `desired_state` is NEVER written by the gate. Retention: prune `node_disk_events` rows older than 30 days on the ingest path (as node_host_metrics does). Pinned contracts (C2/C3/C4/C5 build against these in parallel — do not change them): §3.1 `POST /api/agent/disk-reclaims` (agent auth, NODE_TOKEN; node id is the caller's). Body: {"reclaim_id": "uuid4 idempotency key (UNIQUE; replay = 200 no-op)", "started_at", "finished_at" (ISO-8601 UTC), "trigger": "scheduled | warn | converge", "used_pct_before": 81.2, "used_pct_after": 63.0, "freed_bytes": {"ci_images","build_cache","stopped_containers","orphan_volumes","runner_versions","runner_temp","plugin_dedup","orphan_users"}, "engines": [{"user","uid","bytes_before","bytes_after","volumes_bytes_after","skipped": "busy|null","error"}], "removed_users": [...], "remaining_bytes": {"plugin_root": 0}, "errors": [...]} Every key after freed_bytes is optional. freed_bytes keys are a CLOSED set (unknown key → 422); a producer sends only its own keys. Event row stores used_pct = used_pct_after; used_pct_before, engines, removed_users, remaining_bytes go into `detail`. §3.2 `node_disk_events(id bigserial PK, node_id uuid NOT NULL FK nodes ON DELETE CASCADE, kind text NOT NULL CHECK in ('reclaim','pause','resume'), occurred_at timestamptz NOT NULL DEFAULT now(), trigger text NULL CHECK in ('scheduled','warn','converge'), used_pct real NULL, freed jsonb NOT NULL DEFAULT '{}', detail jsonb NOT NULL DEFAULT '{}', reclaim_id uuid NULL UNIQUE)`, index (node_id, occurred_at DESC). Additive migration. §3.3 Console API: `GET /api/nodes/{id}/disk-events?limit=50` → {"events": [{"id","at","kind","trigger","used_pct","freed_bytes":{...},"total_freed_bytes","source_node":{"id","name"},"detail":{"used_pct_before","engines","removed_users","remaining_bytes"}}]}, newest first (Planner addition, additive: `detail` is served so the console can show per-engine sizes, removed users and plugin-root size — the evidence C6 reads; `detail` carries only keys the report sent); for a co-located vnode also returns its HOST's `reclaim` rows (source_node = host). Node list + node detail payloads gain "disk_gate": {"state": "open"|"paused", "since": ISO, "used_pct": number|null} (latest fresh reading). Gha pool payload gains "paused_by": "operator"|"disk"|null. Same view permission as the node page / Metrics tab. §3.4 The observe response gains "disk_warn_pct": node_host_metrics.WARN_PCT (no second copy of thresholds anywhere — N5). Docs (single-home, each in its existing home): `launch_disk.py` docstring (the fold); `docs/policies/fleet-task-lifecycle.md` wait-reason list if it enumerates them; `jaah_scheduler.py:88-90` comment pointing at `launch_disk.py` (vnode cap still not a reason, N1). Start at: apps/api/app/{launch_disk,dispatch_wait,fleet_observe,node_host_metrics,models,alert_record}.py, routes/{agent,nodes,gha_runners,fleet}.py, lease_drain.py; new apps/api/app/node_disk_events.py; one migration. Follow docs/standards/coding-standards.md (entity shape, additive migrations, revision IDs). Credentials & reach: Unit/integration tests, `./dc check full`: the 7.x CP tests, 11.2 — nothing beyond the worktree — **verified**: needs nothing. (Live synthetic-fill and console proofs of AC-7.x / 8.3 belong to child C6.) Out of scope: Do NOT touch node-side code (apps/jaahd, packaging, infra/host — children C2/C3/C4) or the web console (child C5). Do NOT change WARN_PCT/CRITICAL_PCT values or add a second pair of disk thresholds (N5). Do NOT convert the vnode disk cap watchdog (vnodehelper/usage.py, jaah_scheduler.py note) into a limit; no filesystem/project/user/Docker quotas (N1, D2). No storage sizing from the node layout, no hosting product/picker/EBS changes (N2, Task 5737). No SSM, IAM or AWS tag work (N4, Task 5725). Never write the operator's gha `desired_state` from the gate; never stop a busy runner/kill a running job (N7). Do not change the host-level ephemeral gha slot model (N3). Do not delete the "system/Node maintenance" cron (child C6). No feature flag — every mechanism ON by default (R11). Done means: - Hysteresis table (R7) — unit tests over the transition: open+≥90 → paused (+pause event, +alert, +enqueue_apply poke); paused+80 → stays paused (AC-7.4); paused+<75 → open (+resume event, +poke) (AC-7.3); observation with no/unreadable disk keeps state in both directions; open + stale reading stays open (fail open). - AC-7.1 — test: a ready task targeted at a disk-paused node is neither launched nor resumed (every reader of `launch_disk` — lease_drain dispatch and fleet_allocation, including the resume path), and its wait reason is the `dispatch_wait.disk_paused` sentence naming the usage %. - AC-7.2 / AC-7.5 — tests on the agent desired endpoint: pool running + node paused → served state `paused`, `paused_by: 'disk'`, busy slots not in the stop list; pool operator-paused + node resumes → served state still `paused`, `paused_by: 'operator'`; `gha_runner_pools.desired_state` unchanged by any gate transition. - AC-7.6 / AC-8.3 — test: two nodes (co-located vnodes) pausing in the same minute produce two pause events AND two `alerts` rows (dedup identity carries the node). - AC-7.7 — test: a node that is both paused and below the GB floor yields exactly ONE wait reason (the pause sentence); `launch_disk.py` docstring states how the % gate and GB floor relate; grep shows no other reader computing the disk launch decision. - §3.1 — tests: valid report → one `reclaim` event row with freed/detail split as pinned; replay of the same reclaim_id → 200, no second row; unknown freed_bytes key → 422; wrong/no agent auth → refused. - §3.3 — tests: GET disk-events newest-first with total_freed_bytes and `detail` (used_pct_before, engines, removed_users, remaining_bytes as reported); a vnode's list includes its host's reclaim rows with source_node = host; node list/detail carry `disk_gate`; gha pool payload carries `paused_by`; view permission same as the node page. - §3.4 — test: observe response carries `disk_warn_pct` equal to `node_host_metrics.WARN_PCT`. - Retention — test: events older than 30 days pruned on ingest; pruning never changes `nodes.disk_gate`. - Docs: `launch_disk.py` docstring states the fold; `jaah_scheduler.py:88-90` comment points at `launch_disk.py` (vnode cap still not a reason); `docs/policies/fleet-task-lifecycle.md` wait-reason list carries the disk-paused reason if it enumerates reasons. - Migration is additive (two releases overlap) and round-trips; `./dc check full` green. - PR merged into develop and the Jaah Deploy run for it succeeded.

Built CP disk gate (hysteresis, CI pause, one launch reason), disk events + reclaim API, pause alert, disk_warn_pct. PR #5015 merged; Jaah Deploy green. API suite 20,889 passed; only full-gate red is a NodeLink test that fails on develop too.