Fix October broken sessions root causes
Task 9440 analysed prod's broken sessions for Oct 18–23 (28 breaks, 12 incidents). The attached report has the evidence and code references. Fix these root causes:
1. Relaunch into the task's own held slot dies after an operator Stop or a runner's task_fail (16 breaks: tasks 7456, 7543, 8212, 8683, 9255). The pane log attached to the "vanished" attempt is the predecessor's.
- Fence kill_runner intents on launch_session_id / slot generation, not task id alone (apps/api/app/fleet_verdicts.py build_kill_intent; apps/api/app/routes/tasks/slots.py). An intent minted for a predecessor must never kill a successor launched into the same slot.
- Lease drain relaunches into a held slot only after the node reports the predecessor gone.
- Do not auto-retry a runner's own task_fail when it follows an answered question round; land it system_failed with the runner's reason.
- A vanished incident records the node, the launch id and whether the pane log is the predecessor's.
2. A retry returns to the node that just failed the launch (7 breaks; 7789 failed 4× on h5-3 across 3 accounts).
- launch_failed counts toward the per-node skip (apps/api/app/launch_failures.py).
- When one node fails the same task's launch across different accounts, refund the attempt.
3. Silent stalls are never recorded (8683 sat active 27 h 35 m, 9255 1 h 4 m, after "Runner started" with no session).
- Add a task-level silence detector: an active task with no session attached, heartbeat or transcript write for ~20 min records a break and is reclaimed.
4. The merge lane retries a forge auth rejection ("Invalid username or token", task 7765) until its attempts run out.
- Treat it as a credential blocker: one fresh credential mint and one retry in place, then stop re-dispatching and alert the operator, naming the credential.
5. Misclassified breaks.
- Break classification on the settle/transcript path skips background-subagent task-notification records (apps/jaahd/hooks/break_cause.py), as rs_stopfailure already skips agent_id events.
- Gemini "This client is no longer supported for Gemini Code Assist for individuals" becomes a terminal cause, not auth_expired.
Out of scope: the broken-session collection UI; purge/archival of old incidents; dev control plane; investigating h5-3's hardware/OS beyond what item 2 needs; any other break causes.
Done means:
- Each item ships with a test that reproduces its failure first: a kill minted for launch A does not kill launch B in the same slot; a second launch_failed on node N places the retry on another node; an active task silent past the threshold records a break; a lane auth rejection does not re-dispatch more than once; a subagent limit notification does not classify the session as usage_limit; the Gemini sentence maps to the terminal cause.
- After deploy, `tools/broken_sessions.py list` on prod shows no new held-slot "vanished" incidents following a Stop→Run or task_fail over a 5-day window.
Required checks: ./dc check full
Subtasks
- [business_analysis] Fix October broken sessions root causes PRD settled: 9 of 10 fix asks in scope, each with a test oracle that fails on develop today (5a and 5b reproduced). 1c already shipped in Task 9264 (tests pass). New causes: stalled and client_unsupported (terminal). The 5-day prod check runs after deploy. Memory: none applied.
- [architecture] Fix October broken sessions root causes 7 independent children: C1 kill fence on slot generation; C2 relaunch waits on node-reported predecessor gone; C3 vanished-incident evidence; C4 launch_failed skip + cross-account refund; C5 stalled detector (+VM); C6 lane forge-auth blocker; C7 classifier fixes. Memory: none applied.
- [plan] Fix October broken sessions root causes Split by Architecture boundary: 7 independent coder children (C1–C7) in one phase "Root-cause fixes", no depends_on. R1c needs no child because Task 9264 already shipped it. AC-D2 stays deferred to Finish; it sits in C3, and C1/C2 point to it. Acceptance tightened with the critic's minor cases. Memory: none applied.
- Fence kill intents on slot generation kill_runner intents now carry the hold's generation; the node retires a predecessor's kill as stale instead of killing the relaunch. Kill suppression is per generation. Tests fail on develop first; quick gate green; PR #6766 queued to the merge lane. Memory: none applied.
- Relaunch waits for predecessor gone Own-slot relaunch now waits (uncharged, non-blocking) until the node reports the predecessor gone; plan + take-time gate, 7 new tests red on develop first, quick gate green. PR #6765 queued to merge lane. AC-D2 deferred to Finish. Memory: none applied.
- Record launch evidence on vanished incidents Vanished/launch_failed incident bodies now record node (name (uuid), or uuid (shared node)), launch id (dispatch_id gen N) and predecessor-pane yes/no/unknown. 10 tests red on develop, green after; quick gate green. PR #6762 queued to the lane. AC-D2 deferred to Finish. Memory: none applied.
- Launch-failure node skip and refund PR #6768 queued in merge lane: launch_failed counts toward per-node skip for the 24h window (yields if node is the task's only live one); cross-account launch failure on a node refunds the attempt (capped, noted). Quick gate green; 14 tests red on develop first. No UI surface affected. Memory: none applied.
- Task-level silent stall detector Stall detector shipped: active task silent 20 min (no session/sighting/transcript write) records one `stalled` break and is reclaimed; starved nodes and provisioning VMs excluded. Quick gate + changed-area tests green; review lane passed; PR #6770 queued for full gate. Memory: none applied.
- Merge-lane forge auth credential blocker Merge-lane forge auth rejection is now a credential blocker: one fresh mint + one in-place retry, second refusal → system_failed naming the credential + one alert, no re-dispatch. Quick gate green, red-on-develop proven, review lane pass; PR #6769 queued (lane conflict rebased). Deferred: broker refusal set is in-memory. Memory: none applied.
- Fix subagent and Gemini break classification Subagent <task-notification> records no longer classify breaks; Gemini "client no longer supported" is new terminal cause client_unsupported (no auth verdict, readable label). New tests red on develop, green on branch; quick gate green. PR #6764 queued (merged with #6762).
- [finish] Fix October broken sessions root causes All 7 coders (9464–9470) done, PRs #6762/64/65/66/68/69/70 merged to develop; 5 root causes covered (1c already shipped via Task 9264). No follow-ups. Owed: prod 5-day oracle after master promotion. Memory: none applied.
Artifacts
Attachments
Activity
- Created
- Runner started
- Ready
- Ready
- Ready
- Runner started
- Runner ended (done)
- Runner started
- Runner ended (done)
- Runner started
- Draft
- Draft
- Draft
- Draft
- Draft
- Draft
- Draft
- Draft
- Ready
- Ready
- Ready
- Ready
- Ready
- Ready
- Ready
- Ready
- Runner ended (done)
- Runner started
- Runner started
- Runner started
- Runner started
- Runner started
- Runner started
- Runner started
- Runner ended (done)
- Runner ended (done)
- Runner ended (done)
- Runner ended (done)
- Runner ended (done)
- Runner ended (done)
- Runner ended (done)
- Runner started
- Runner ended (done)
- Runner ended (done)