NodeLink provisioning progress and retries

Epic: Shared account tokens — make the NodeLink flow (plan 6407; PRD 6405, architecture 6406). Items 3, 4 and 5 of 10. Context (binding): PRD decisions D1–D8 and architecture decisions A1–A6 are settled; do not reopen them. Every capability is on by default for an org whose plan has the NodeLink term, with no new flag. "Live stack" means app.dev.jaah.ai after your PR's Jaah Deploy succeeds; verify there in a browser. A check that the dev tenant TimeSaver2 cannot support is recorded as deferred, with the reason, in the PR body (A5). Each item is either shipped or rejected with a reason in the PR body. ## Requirements (verbatim) 3. "NodeLink Storage tab: while the storage is `provisioning`, the page should poll (or the Provision call should wait) and show a progress line (file system → mount target → access point → policy) instead of "The storage is not ready yet. Provision this NodeLink again." and three manual presses over two minutes." 4. "NodeLink provision: right after the access point is created the NodeLink flipped to `failed` with "the access point is deleted, not available" although AWS already described it as `available`; Retry succeeded. Treat an immediate `deleted`/unknown state as transient and re-describe once before failing (`apps/api/app/nodelinks.py`, the AP state loop)." 5. "NodeLink Storage › Create new: the AWS credential picker shows nothing when the org has no AWS credential; add an inline hint linking to Secrets › Add credential." Decision A2: the **server continues** provisioning. The page only polls. ## Approach (architecture, C2) - **Semantics.** `POST …/provision` becomes a *request*. It runs one synchronous pass, as today; if not finished, the row is left `provisioning`. That now includes "store still being built", which today returns `pending`, so `pending` again means "not requested". `pending_reason` `STORE_NOT_READY` stops surfacing for a requested row. - **Cron.** Add `nodelink-provision` (every 15 s) to `apps/api/app/cron_system_activities.py`, following the existing system-cron pattern. Each pass: - Selects `nodelinks WHERE status='provisioning'` and runs `nodelinks.provision(session, row, actor_id=None)` per row under `nowait` locks; a BUSY row is skipped. - Change `provision`'s `actor_id` (`nodelinks.py:708`) to `Optional[UUID]`. - A system pass keeps `updated_by_id` as the requester's and writes no audit row. - When a row reaches `ready`, send its attached nodes `nodelink.request_converge`. - **Time bound (3c).** No new column. A store still not `available` 30 min after the FS `CreationTime` that AWS returns fails with "AWS didn't finish creating the file system in 30 minutes." - **Progress (3a).** `NodeLinkDetailOut` gains a derived field, never stored: - Shape: `progress: [{"step": "file_system"|"mount_target"|"access_point"|"policy", "state": "done"|"current"|"waiting"|"failed"}] | null`. - It is null unless status ∈ {`provisioning`, `failed`}. - `file_system` is done when `store.file_system_id` is set and the store is past FS wait. - `mount_target` is done when `store.mount_target_id` is set and the store is `ready`. - `access_point` is done when `row.access_point_id` is set and the AP is available. - `policy` is done when `row.in_policy`. - An adopted (or, later, managed) store marks the first two done. - On `failed`, the step that was current carries `failed`. - **Progress web.** Add one horizontal line component, `apps/web/src/pages/nodes/nodeLinkStore/ProvisionProgress.tsx`, on the Storage tab and the NodeLink page header. It wraps to a vertical list under 600 px (follow the breakpoint idiom in `components/wizard/WizardSteps.tsx`). - In `NodeLinkEditPage.tsx:82-150`, **delete** the auto re-press loop. Keep `usePoll(15s)` on `getNodeLink` while `provisioning`/`deleting`. - Delete "Still creating. Press Provision again." (`NodeLinkStorageTab.tsx:264-267`). - A failure keeps the AWS text and Retry. - **Item 4.** In `_provision`'s AP loop (`nodelinks.py:795-805`), a `None`/`deleted`/unknown state re-describes ONCE after `AP_POLL_SECONDS` before raising. If the re-describe returns available/creating/updating, the loop continues (to `ready`, or to `provisioning` at the deadline, where the cron resumes). If it returns deleted or missing again, the row fails with the existing text. - **Item 5.** In `NodeLinkStorageTab.tsx::credentialField` (`:248-260`, used at `:331`, `:350`), when `fetchAwsCredentialOptions` returns `[]`, show the inline hint "No AWS credential." with a link **Add credential** → `/credentials/new?type=aws&return=<current url>`. Make the Add credential form honour `type`/`return` if it does not already. On return, the options re-fetch. - **Pinned seam (consumed by C1a, C3).** Extract the Storage tab's source controls into a shared `StorageSourceFields` component with props `{orgStores, value, onChange, managed: {available, location} | null}`; this child passes `managed: null`. Once requested, provision returns `provisioning`, never `pending`. The `progress` shape is exactly as above. ## Acceptance criteria (verbatim from PRD) ### 3. Provisioning progress - 3a. While `pending`(after a provision request)/`provisioning`, the Storage tab and NodeLink page show one progress line with the four steps, each done/current/waiting, updating without any press. → verify on dev: create new storage; watch steps advance to `ready` with zero Provision presses. - 3b. "The storage is not ready yet. Provision this NodeLink again." and "Still creating. Press Provision again." are never shown in a non-failed state. → verify: grep web for the strings in non-error paths; observe on dev. - 3c. A genuine failure still shows the AWS reason and Retry. Polling stops at `ready`/`failed`; no indefinite loop (the existing cap may be replaced by a time bound ≥ the worst-case EFS creation time). - 3d. Applies to every provisioning route (create new, existing store, Jaah-managed, and the item-1 dialog). (This child covers create new and existing store. Jaah-managed is verified in the C1a child and the item-1 dialog in the C3 child; both reuse this engine.) ### 4. Transient access-point state - 4a. A `deleted`/missing/unknown AP state observed within the wait is re-described once before failing; if the re-describe returns available/creating, provisioning continues. → verify: API unit test faking describe returning `deleted` then `available` → row `ready`; faking `deleted` twice → `failed` with the existing message. ### 5. Empty AWS credential picker - 5a. With zero AWS credentials, Create new shows an inline hint with a link to Secrets › Add credential (type AWS preselected if supported); returning shows the new credential. → verify on dev in an org with no AWS credential. - 5b. Same hint wherever the AWS credential picker appears (Create new, existing store). With ≥1 credential, no hint. Credentials & reach: - Visual checks, items 1, 3–10 on dev: `jaah/debug/main/operator-login` (operator; switches to tenants) on app.dev.jaah.ai. **Verified**: `task_credential_get` succeeded, and browser sign-in to app.dev.jaah.ai (1.2.535) succeeded from node h6-3. - Items 1e, 3a, 3d, 5a live (a tenant AWS credential with EFS/EC2 rights for Create new): dev tenant TimeSaver2, console credential "TimeSaver2 AWS" (seen in the dev Secrets list; its rights were not probed). **Deferred** (A5): the coder uses it as it is; if it lacks rights, that check is deferred in the PR body. - All automated tests: worktree only (`./dc check full`); needs nothing beyond the worktree. Out of scope: - Epic: "Out of scope: any change to the NodeLink data model beyond what item 2's defaults need, the mount helper, tailnet ACLs, or the EFS policy model." - Do not modify `packaging/jaah-shared-mount`, tailnet/headscale ACLs, or the EFS access-point policy model (`write_policy` semantics, uid/gid 3000, `tls,noexec,nosuid,nodev`). - No NodeLink kind, managed flag or shared-account flag. No code path keyed on the accounts path beyond `ACCOUNTS_MOUNT_PATH`. - No new billing line (D1). No change to the jaahd token-dir resolver semantics. No change to Org/System credential ownership rules. - This child: no new DB column (no `provision_started_at`); no Jaah-managed source (C1a adds it); no share-subscriptions dialog (C3); no client-side re-press loop that dies when the tab closes. Done means: - 3a, 3b, 3c and 3d (create new + existing store) as above. Steps advance on dev with zero Provision presses and the tab may be closed mid-way: the row still reaches `ready` (the server cron, not the page, drives it). The progress line is legible at 360 px (vertical list, no truncated step labels). - 4a: API unit tests as specified (deleted→available ⇒ `ready`; deleted twice ⇒ `failed` with the existing message), plus a cron test that a `provisioning` row advances with no request and a BUSY row is skipped. - 5a and 5b as above. The hint and link are readable at 360 px, and returning from Add credential shows the new credential without a manual reload. - `StorageSourceFields` is extracted and used by the Storage tab (no copy left behind). - `./dc check full` green; Jaah Deploy succeeded; screenshots of progress and the hint on dev, or a deferral reason (A5) in the PR body.

NodeLink provisioning is server-driven (nodelink-provision cron) with a 4-step progress line, an AP re-describe-once, an Add-credential hint and StorageSourceFields. Full gate green; PR #5517 merged, dev deploy green. Live Create new and hint checks deferred (A5, no tenant has NodeLink); verified locally. mem-2 used; mem-18205 proposed.