Merge-lane forge auth credential blocker

## Epic requirement (verbatim, Task 9453 body) Task 9440 analysed prod's broken sessions for Oct 18–23 (28 breaks, 12 incidents). The attached report has the evidence and code references. Fix these root causes: 1. Relaunch into the task's own held slot dies after an operator Stop or a runner's task_fail (16 breaks: tasks 7456, 7543, 8212, 8683, 9255). The pane log attached to the "vanished" attempt is the predecessor's. - Fence kill_runner intents on launch_session_id / slot generation, not task id alone (apps/api/app/fleet_verdicts.py build_kill_intent; apps/api/app/routes/tasks/slots.py). An intent minted for a predecessor must never kill a successor launched into the same slot. - Lease drain relaunches into a held slot only after the node reports the predecessor gone. - Do not auto-retry a runner's own task_fail when it follows an answered question round; land it system_failed with the runner's reason. - A vanished incident records the node, the launch id and whether the pane log is the predecessor's. 2. A retry returns to the node that just failed the launch (7 breaks; 7789 failed 4× on h5-3 across 3 accounts). - launch_failed counts toward the per-node skip (apps/api/app/launch_failures.py). - When one node fails the same task's launch across different accounts, refund the attempt. 3. Silent stalls are never recorded (8683 sat active 27 h 35 m, 9255 1 h 4 m, after "Runner started" with no session). - Add a task-level silence detector: an active task with no session attached, heartbeat or transcript write for ~20 min records a break and is reclaimed. 4. The merge lane retries a forge auth rejection ("Invalid username or token", task 7765) until its attempts run out. - Treat it as a credential blocker: one fresh credential mint and one retry in place, then stop re-dispatching and alert the operator, naming the credential. 5. Misclassified breaks. - Break classification on the settle/transcript path skips background-subagent task-notification records (apps/jaahd/hooks/break_cause.py), as rs_stopfailure already skips agent_id events. - Gemini "This client is no longer supported for Gemini Code Assist for individuals" becomes a terminal cause, not auth_expired. Out of scope: the broken-session collection UI; purge/archival of old incidents; dev control plane; investigating h5-3's hardware/OS beyond what item 2 needs; any other break causes. Done means: - Each item ships with a test that reproduces its failure first: a kill minted for launch A does not kill launch B in the same slot; a second launch_failed on node N places the retry on another node; an active task silent past the threshold records a break; a lane auth rejection does not re-dispatch more than once; a subagent limit notification does not classify the session as usage_limit; the Gemini sentence maps to the terminal cause. - After deploy, `tools/broken_sessions.py list` on prod shows no new held-slot "vanished" incidents following a Stop→Run or task_fail over a 5-day window. Required checks: ./dc check full (Item 1c — task_fail after an answered blocker round lands system_failed — already shipped in Task 9264 (`58b8ffd9b`); NOT part of any child.) Settled inputs: the full PRD (Task 9451) and Architecture (Task 9452) are the epic's `prd` / `architecture` artifacts; this child is the Architecture's section named below. Evidence report: attachment 9076_report.html (Task 9440). Every new test must FAIL on today's develop first — cite the red run in the PR. --- # Child C6 — Merge-lane forge auth credential blocker ## Scope (this child) Epic item 4 (PRD R4; Architecture §C6, decisions D9, D10). Verbatim ask: "The merge lane retries a forge auth rejection ("Invalid username or token", task 7765) until its attempts run out. — Treat it as a credential blocker: one fresh credential mint and one retry in place, then stop re-dispatching and alert the operator, naming the credential." PRD R4 (verbatim): A forge authentication rejection in a merge-lane run (git/forge stderr naming the refusal: `Invalid username or token`, `Authentication failed`, HTTP 401/403 auth refusal from the forge — exact phrase set is the Architect's, each phrase self-identifying) is a **credential blocker**: 1. exactly **one** fresh credential mint (cache bypassed) and **one** retry of the failed operation **in place** (same run, same slot — no re-dispatch); 2. if the retry is also rejected, the lane task is **not re-dispatched again**: it lands `system_failed` with a reason naming the credential, and **one operator alert** is raised (existing alert/notify surface used by the merge lane, `merge_lane.py::_alert` / `record_alert`) naming the credential — its connection/credential name, owner and forge/repo — **never its value**. - Decided: scope is the merge lane only; coder runners' push failures are a deferred follow-up. - Decided: the alert is deduped per credential while it stays unfixed. ## Approach (Architecture, settled) - D9: the signal is git's own `erase` call to the credential helper (sent only after the server refused the supplied login), plus a 401 seen by `jaah-forge` — not phrase-matching runner text (the naive version; do not ship it). - Node: `git_credential_helper.main`'s `erase` arm (`apps/jaahd/git_credential_helper.py:328-336`) keeps its `fresh=True` re-read (the one mint; the broker caches the fresh login so git's next `get` is served it) and, through the broker (`apps/jaahd/cred_broker.py`, task attached as for `git-token`), sends new vocabulary verb `forge_auth.rejected {host, repo_path}`. The `store` arm (`:337-339`) sends `forge_auth.accepted` only when the broker holds an unresolved `rejected` for that task and identity. `forge_cli` retries its own forge-API call exactly once on HTTP 401 after a fresh login, then reports. Fire-and-forget: a report failure never fails git. - CP: `POST /api/agent/tasks/{task_id}/forge-auth-rejected` and `/forge-auth-accepted` in `apps/api/app/routes/agent/task_secrets.py`, body `{"host": str, "repo_path": str|None}` → `{"count": int, "terminal": bool}`, node-token auth with the git-token caller-binds-task rule. Extract `git_login_source(session, task) -> CredentialSource(id, name, owner, forge, repo)` from the git-token resolution, shared by both routes. Ignore non-`merge_lane.LANE_PERSONA` tasks. Write a `task_note` cause `forge_auth_rejected`/`forge_auth_accepted` with `data.credential=<id>`. On this attempt's second rejection (notes since latest claim): `end_attempt(final=True)` from `active`, or direct `system_failed` from `ready`; reason `forge refused credential "<name>" (<owner>, <forge>/<repo>) twice — reconnect it`; kill minted as usual; `record_alert(kind="error", issue=f"forge-auth:{credential_id}", task_id=…, reason=same)` (`apps/api/app/alert_record.py:112-126`). - D10: `merge_lane.unresolved_forge_rejection(session, task_id) -> Optional[CredentialSource]` (newest forge-auth note of this attempt is a rejection) is read by the report route, `fail_task` (`routes/agent/task_terminal.py:1040-1047`) and `runner_settle.reclaim_active_task`: such an attempt ends `final=True` with the credential reason and alert — never re-dispatched. - Charter: one line in the lane persona's prompt text in `merge_lane.py` (not the runner charter): retry an auth rejection once in place; after the second, stop. - Security surface (credentials): Full Auto — clear it with the autonomous review lane. No token value in any reason, alert, note or log. ## Credentials & reach | Oracle | Credential / identity / environment | Status | |---|---|---| | AC4 (unit and API tests) | Nothing beyond the worktree's own `./dc` stack | verified — runs in any leased worktree, no roster credential needed | | AC-RC `./dc check full` | Merge lane's worktree, no credential | verified — the lane runs it per project config | Out of scope (PRD §7, verbatim) — do NOT touch: - MUST NOT change the broken-session collection UI (console pages listing runner-break incidents) beyond the incident **body text** R1d requires. - MUST NOT add purge/archival of old incidents or change `tools/broken_sessions.py purge` / `runner_breaks.stale_escalations`. - MUST NOT change dev control plane config or data; no dev-only behaviour. - MUST NOT investigate or change h5-3's hardware/OS, node packaging or host setup beyond what item 2's placement/refund needs. - MUST NOT add or reclassify any other break cause (e.g. `timeout`, `api_error`, `crashed`, `unknown` rules) beyond `stalled` and `client_unsupported`; the unexplained 8569 break is not addressed. - MUST NOT re-implement 1c (Task 9264 shipped it) or widen it to non-blocker interview rounds. - MUST NOT change coder-runner push/credential handling (merge lane only for item 4). Epic out of scope: the broken-session collection UI; purge/archival of old incidents; dev control plane; investigating h5-3's hardware/OS beyond what item 2 needs; any other break causes. No feature flag, setting or operator toggle may gate the fix (PRD: on by default once deployed). Architecture boundary: merge-lane tasks only; the CP ignores non-lane reports (coder-runner push auth is a deferred follow-up). Done means: - AC4: A lane run rejected by the forge: one fresh mint + one in-place retry; on second rejection the task is `system_failed` with the credential named, one alert naming the credential exists, and the task is **not** re-dispatched (attempt count unchanged after further drain passes). No credential value appears in reason, alert or logs. — jaahd test for the helper/lane retry + API test for the terminal + alert → pass (red on develop first). Cases: jaahd `erase` → one fresh read + one `rejected` report, next `get` served the fresh login (cache bypassed exactly once), `store` → one `accepted` report; jaahd `forge_cli` 401 → exactly one retry then a report; API first report → note only, task active; first report then `/fail` → `system_failed`, alert, no re-dispatch; rejected→accepted→`/fail` → ordinary retry; second report → `system_failed` naming the credential, one alert, `dispatch_attempts` unchanged after two more drain passes; non-lane task → no-op; sentinel token string absent from reason, alert and logs. - Epic done-means (item 4): a lane auth rejection does not re-dispatch more than once — the AC4 test above. - AC-RC: Full gate green — `./dc check full` (merge lane) → green. Required checks: ./dc check full

Merge-lane forge auth rejection is now a credential blocker: one fresh mint + one in-place retry, second refusal → system_failed naming the credential + one alert, no re-dispatch. Quick gate green, red-on-develop proven, review lane pass; PR #6769 queued (lane conflict rebased). Deferred: broker refusal set is in-memory. Memory: none applied.