Task-level silent stall detector

## Epic requirement (verbatim, Task 9453 body) Task 9440 analysed prod's broken sessions for Oct 18–23 (28 breaks, 12 incidents). The attached report has the evidence and code references. Fix these root causes: 1. Relaunch into the task's own held slot dies after an operator Stop or a runner's task_fail (16 breaks: tasks 7456, 7543, 8212, 8683, 9255). The pane log attached to the "vanished" attempt is the predecessor's. - Fence kill_runner intents on launch_session_id / slot generation, not task id alone (apps/api/app/fleet_verdicts.py build_kill_intent; apps/api/app/routes/tasks/slots.py). An intent minted for a predecessor must never kill a successor launched into the same slot. - Lease drain relaunches into a held slot only after the node reports the predecessor gone. - Do not auto-retry a runner's own task_fail when it follows an answered question round; land it system_failed with the runner's reason. - A vanished incident records the node, the launch id and whether the pane log is the predecessor's. 2. A retry returns to the node that just failed the launch (7 breaks; 7789 failed 4× on h5-3 across 3 accounts). - launch_failed counts toward the per-node skip (apps/api/app/launch_failures.py). - When one node fails the same task's launch across different accounts, refund the attempt. 3. Silent stalls are never recorded (8683 sat active 27 h 35 m, 9255 1 h 4 m, after "Runner started" with no session). - Add a task-level silence detector: an active task with no session attached, heartbeat or transcript write for ~20 min records a break and is reclaimed. 4. The merge lane retries a forge auth rejection ("Invalid username or token", task 7765) until its attempts run out. - Treat it as a credential blocker: one fresh credential mint and one retry in place, then stop re-dispatching and alert the operator, naming the credential. 5. Misclassified breaks. - Break classification on the settle/transcript path skips background-subagent task-notification records (apps/jaahd/hooks/break_cause.py), as rs_stopfailure already skips agent_id events. - Gemini "This client is no longer supported for Gemini Code Assist for individuals" becomes a terminal cause, not auth_expired. Out of scope: the broken-session collection UI; purge/archival of old incidents; dev control plane; investigating h5-3's hardware/OS beyond what item 2 needs; any other break causes. Done means: - Each item ships with a test that reproduces its failure first: a kill minted for launch A does not kill launch B in the same slot; a second launch_failed on node N places the retry on another node; an active task silent past the threshold records a break; a lane auth rejection does not re-dispatch more than once; a subagent limit notification does not classify the session as usage_limit; the Gemini sentence maps to the terminal cause. - After deploy, `tools/broken_sessions.py list` on prod shows no new held-slot "vanished" incidents following a Stop→Run or task_fail over a 5-day window. Required checks: ./dc check full (Item 1c — task_fail after an answered blocker round lands system_failed — already shipped in Task 9264 (`58b8ffd9b`); NOT part of any child.) Settled inputs: the full PRD (Task 9451) and Architecture (Task 9452) are the epic's `prd` / `architecture` artifacts; this child is the Architecture's section named below. Evidence report: attachment 9076_report.html (Task 9440). Every new test must FAIL on today's develop first — cite the red run in the PR. --- # Child C5 — Task-level silent stall detector ## Scope (this child) Epic item 3 (PRD R3; Architecture §C5, decisions D7, D8). Verbatim ask: "Add a task-level silence detector: an active task with no session attached, heartbeat or transcript write for ~20 min records a break and is reclaimed." PRD R3 (verbatim): A task-level silence detector runs on an existing CP cron. A task in state `active` for which **all three** hold for ≥ **20 minutes** (one named constant, the single home of the value) is silent: 1. no session attached (task has no `runner_session_id` / slot sensed session not live), 2. no heartbeat (no node sighting of the runner: `worktrees.last_seen_live` / equivalent older than the threshold), 3. no transcript write for the task's session in that time. A silent task **records a break** (one incident per task, deduped like every other cause) and **is reclaimed** through the existing reclaim path (attempt spent, task to `ready`, or `system_failed` on the last attempt; slot kept per worktree-pool policy). - Decided: new CP-observed cause **`stalled`**, added to `runner_breaks.CAUSES` (CP-only, like `vanished`/`launch_failed`, so not added to the node mirror). - States excluded: `awaiting_response`, `paused`, `ready`, `draft`, `done`, `system_failed` are never judged. A task `active` < 20 min since its claim is never judged. A runner that is alive and writing transcript is never judged silent, however long it runs. - The 20-minute window starts from the latest of: claim/launch time, last heartbeat, last transcript write — so a freshly relaunched task is not reclaimed for its predecessor's silence. ## Approach (Architecture, settled) - Node: slot observation adds `age_transcript_write_secs: Optional[int]` — age of the mtime of the slot's live session transcript (`break_cause.project_dir(cwd=slot path, config_dir=slot account dir)` + `session_transcript_path`), `None` when nothing resolves (ages-only rule). - CP: `ObserveSlotIn` (`apps/api/app/fleet_observe.py:222-260`) accepts it; `slot_update_rows` (`:1163-1234`) writes `transcript_written_at = observed_at − age` only when present, else keeps the prior value (old nodes write nothing). - ONE additive migration: nullable `worktrees.transcript_written_at` and nullable `microvms.last_sighted_at`. Revision id from the repo's own tool per `docs/standards/coding-standards.md` (migration revision IDs). This is the only migration in the epic. - Detector: `task_pipeline.reclaim_silent_active_tasks()` called from `task_pipeline_job.sweep_once` (`apps/api/app/task_pipeline_job.py:36`). `SILENCE_THRESHOLD = timedelta(minutes=20)` lives in `runner_liveness.py` next to `OBSERVATION_FRESH_SECONDS` (one home). One query: `active` tasks holding a static slot where `greatest(task.github_synced_at, wt.last_seen_live, wt.transcript_written_at) < now() − SILENCE_THRESHOLD` AND `NOT (wt.session IS TRUE AND wt.observed_at >= now() − SILENCE_THRESHOLD)`. Second arm for MicroVM runs (no static slot, newest VM not ended): `greatest(task.github_synced_at, vm.last_sighted_at) < now() − SILENCE_THRESHOLD`; the CP stamps `microvms.last_sighted_at = now()` in the existing VM-report ingest (`microvm_lifecycle`) whenever a report shows that VM `running`. - Reclaim: each hit → `task_pipeline_cp.reclaim_stalled(task_id)` = `_reclaim_active(task_id, observed_by="stall-detector", cause=CAUSE_STALLED)`; `_reclaim_active` gains `*, cause: Optional[str] = None` (`None` keeps today's behaviour — pinned seam with C3). `reclaim_active_task`'s reason selection (`runner_settle.py:585`) gains a `stalled` branch: "the runner went silent for 20 min (no session, heartbeat or transcript write), and every attempt is spent" — minutes rendered from `SILENCE_THRESHOLD`, never hand-written. - Vocabulary: `CAUSE_STALLED = "stalled"` in `runner_terminations` (+ `RUNNER_LOST_CAUSES`), `runner_breaks.CAUSES` (CP-only comment), `CAUSE_LABELS` label "went silent". Also add `stalled` to `runner_terminations.CAUSES` and `RESTART_HINT_CAUSES` as required by `apps/api/tests/test_runner_terminations.py`. Confirm the claim stamps `task.github_synced_at` (else use the claim timestamp that does); note an old node leaves `transcript_written_at` NULL, which drops the transcript guard on that node during skew — accepted, since no detector exists today. C7 appends `client_unsupported` to the same tuples — file overlap only. Ensure the Sessions end-reason label (`apps/api/app/routes/sessions.py:370-393`, `end_reason_label`) renders "went silent", not the slug. - Risk (false reclaim of a working runner with a wrong session flag): all three timestamps must be old and no fresh observation may report a session; the transcript signal is what prevents the naive "no session for 20 min" version — do not ship that version. ## Credentials & reach | Oracle | Credential / identity / environment | Status | |---|---|---| | AC3, AC-L (unit and API tests) | Nothing beyond the worktree's own `./dc` stack | verified — runs in any leased worktree, no roster credential needed | | AC-RC `./dc check full` | Merge lane's worktree, no credential | verified — the lane runs it per project config | Out of scope (PRD §7, verbatim) — do NOT touch: - MUST NOT change the broken-session collection UI (console pages listing runner-break incidents) beyond the incident **body text** R1d requires. - MUST NOT add purge/archival of old incidents or change `tools/broken_sessions.py purge` / `runner_breaks.stale_escalations`. - MUST NOT change dev control plane config or data; no dev-only behaviour. - MUST NOT investigate or change h5-3's hardware/OS, node packaging or host setup beyond what item 2's placement/refund needs. - MUST NOT add or reclassify any other break cause (e.g. `timeout`, `api_error`, `crashed`, `unknown` rules) beyond `stalled` and `client_unsupported`; the unexplained 8569 break is not addressed. - MUST NOT re-implement 1c (Task 9264 shipped it) or widen it to non-blocker interview rounds. - MUST NOT change coder-runner push/credential handling (merge lane only for item 4). Epic out of scope: the broken-session collection UI; purge/archival of old incidents; dev control plane; investigating h5-3's hardware/OS beyond what item 2 needs; any other break causes. No feature flag, setting or operator toggle may gate the fix (PRD: on by default once deployed). Architecture boundary: do NOT change the epic-level stall alarm (`task_pipeline.py:517,589`) or add a parallel reclaim path; reuse `_reclaim_active`. Done means: - AC3: An `active` task silent (no session, heartbeat, transcript write) for ≥ 20 min records exactly one `stalled` incident and is reclaimed; at 19 min, or with any one signal fresh, or in `awaiting_response`/`paused`, nothing happens. — New API test on the detector's cron function → pass (red on develop first). Cases on a fake clock: 20 min silent → one `stalled` incident + reclaimed; 19 min → nothing; fresh `last_seen_live` / fresh transcript / fresh claim → nothing; stale (> 20 min) observation still `session=true` with no other signal → `stalled`; VM run with no `running` report for 20 min → `stalled`, one reported running → nothing; `awaiting_response`/`paused` → nothing; second pass → still one incident. Plus a jaahd test for `age_transcript_write_secs`. - AC-L (stalled half): The Sessions end-reason filter/label shows a readable label for `stalled` (no raw slug). — API test over `end_reason_label` → pass. - Epic done-means (item 3): an active task silent past the threshold records a break — the AC3 test above. - AC-RC: Full gate green — `./dc check full` (merge lane) → green. Required checks: ./dc check full

Stall detector shipped: active task silent 20 min (no session/sighting/transcript write) records one `stalled` break and is reclaimed; starved nodes and provisioning VMs excluded. Quick gate + changed-area tests green; review lane passed; PR #6770 queued for full gate. Memory: none applied.